Dask Gateway Example

This example shows how to start and manage a Dask cluster on a hub that is deployed with Dask Gateway.

It uses the Microsoft Planetary Computer Hub, but should work on any Hub with Dask Gateway.

We use this job configuration file:

# file: dask-gateway-job.yaml
name: dask-gateway-job
image: mcr.microsoft.com/planetary-computer/python:latest
args:
  - python
  - dask-gateway.py
code: dask-gateway.py

The job runs this python file:

"""
Start a cluster with Dask Gateway, print the dashboard link, and run some tasks.
"""
import dask_gateway
from distributed import wait


def inc(x):
    return x + 1


def main():
    gateway = dask_gateway.Gateway()

    print("Starting cluster")
    cluster = gateway.new_cluster()
    client = cluster.get_client()
    print("Dashboard at:", client.dashboard_link)

    cluster.scale(2)

    futures = client.map(inc, list(range(100)))
    _ = wait(futures)

    print("Closing cluster")
    cluster.close()


if __name__ == "__main__":
    main()

And is submitted with

$ kbatch job submit -f dask-gateway-job.yaml