Spark cluster from your python environment


Install

$ pip install ods

ods package uses staroid python library. Get access token and set STAROID_ACCESS_TOKEN environment variable to configure it.

$ export STAROID_ACCESS_TOKEN="<your access token>"

You can find alternative way to configure Staroid package. Learn more.

Initialize

import ods
# 'ske' is the name of kubernetes cluster created from staroid.com.
# Alternatively, you can set the 'STAROID_SKE' environment variable.
ods.init(ske="kube-cluster-1")

Usage

Create spark session with the default configuration
import ods
spark = ods.spark("my-cluster").session()
Create spark session with 3 worker nodes
import ods
spark = ods.spark("my-cluster", worker_num=3).session()
Create spark session with delta lake
import ods
spark = ods.spark("my-cluster", delta=True).session()

See README for more details.