Spark cluster from your python environment


$ pip install ods

ods package uses staroid python library. Get access token and set STAROID_ACCESS_TOKEN environment variable to configure it.

$ export STAROID_ACCESS_TOKEN="<your access token>"

You can find alternative way to configure Staroid package. Learn more.


import ods
# 'ske' is the name of kubernetes cluster created from
# Alternatively, you can set the 'STAROID_SKE' environment variable.


Create spark session with the default configuration
import ods
spark = ods.spark("my-cluster").session()
Create spark session with 3 worker nodes
import ods
spark = ods.spark("my-cluster", worker_num=3).session()
Create spark session with delta lake
import ods
spark = ods.spark("my-cluster", delta=True).session()

See README for more details.