Dev Guide#
Development setup#
Clone the kbatch
repo onto your local machine and create a virtual environment (conda
, venv
, etc.) with Python 3.9 installed. From this directory, you can pip install the packages needed for development
$ pip install -e ./kbatch[all]
...
$ pip install -e ./kbatch-proxy[all]
NOTE: The
-e
signifies that this is an “editable install”, read more here.
The first pip install command above should’ve also installed pre-commit
which is used to format and lint the code. Run the following command to start using pre-commit
$ pre-commit install --install-hooks
Run pytest
#
To validate your changes against the test suite, run
$ pytest -v ./kbatch
...
$ pytest -v ./kbatch-proxy
Run mypy
#
To perform mypy
type checking, run
$ mypy ./kbatch
...
$ mypy ./kbatch-proxy
To build the docs locally#
To build the docs locally, run
$ cd ./docs
$ make html
kbatch-proxy
development setup#
We don’t have a fully working docker-ized setup, since we (i.e. Tom) don’t know how to do Kubernetes within docker. So the current setup relies on
k3d for Kubernetes
JupyterHub as a regular Python process
kbatch-proxy as a regular Python process
Create a cluster#
$ k3d cluster create ksubmit
Create a Hub#
make sure to npm install configurable-http-proxy
.
$ cd hub
$ jupyterhub
Start kbatch-proxy#
KBATCH_PREFIX=/services/kbatch \
KBATCH_PROFILE_FILE=tests/profile_template.yaml \
JUPYTERHUB_API_TOKEN=super-secret \
JUPYTERHUB_API_URL=http://127.0.0.1:8000/hub/api \
JUPYTERHUB_HOST=http://127.0.0.1:8000 \
uvicorn kbatch_proxy.main:app --reload --port=8050
You’ll might want to log in and create a token at http://localhost:8000/hub/token. The kbatch configure
with that token.
Release process#
These are the current steps that it takes to cut a new release of kbatch
.
Release new version of kbatch
/kbatch-proxy
:#
Update the version in the following files:
Update any relevant docs and merge these changes into the codebase.
Draft a release on the
kbatch
GitHub repo..This triggers the
publish-image.yaml
workflow which:publishes a container image for
kbatch-proxy
to the GitHub container registry.publish
kbatch
andkbatch-proxy
python libraries to PyPI.
Release new version of helm-chart
:#
Once the container image and libraries have been successfully published, update the version of the
helm-chart
to match this new version ofkbatch
:Update any relevant docs and merge these changes into the codebase.
Draft a release on the
helm-chart
GitHub repo.This triggers the
publish-chart.yaml
workflow which:publishes a Helm chart to the GitHub Pages site for this repo.
NOTE: there are known limitations to the pre-release process outlined here.