These are the steps to deploy MinIO, Jupyterhub, BinderHub, and Dask Gateway using the baremetal k8s cluster at CESNET in C-SCALE.
This guide assumes that you have:
- Access to a kubernetes cluster with
kubectl
. - Helm is installed (i.e. the
helm
command works). - Access to configure DNS hostnames (e.g. the EGI Dynamic DNS)
For example, using git clone
, and make sure you are in the correct
workding directory:
cd working/dir/
git clone [email protected]:pangeo-data/pangeo-eosc.git
cd providers/kubernetes/
To deploy Jupyterhub along with Dask Gateway we use Daskhub.
The official helm chart uses CustomResourceDefinitions that are not allowed in the baremetal kubernetes deployment at CESNET. The modified chart under the daskhub/ folder is adapted to avoid that issue.
Please configure the values below accordingly:
- daskhub-secrets.yaml:
token1
andtoken2
must be replaced with a hash generated usingopenssl rand -hex 32
on Linux. - daskhub.yaml:
pangeo-eosc.vm.fedcloud.eu
must be replaced with your DNS hostname.
Use helm
to install DaskHub with the command below:
helm upgrade daskhub daskhub/ \
--install --wait \
--cleanup-on-fail \
--create-namespace \
--namespace c-scale-pangeo-dask \
--values daskhub.yaml \
--values daskhub-secrets.yaml
All going well DaskHub will be available at https://pangeo-eosc.vm.fedcloud.eu.
To get Dask Gateway working with Binder, first we deploy BinderHub and then Dask Gateway.
Due to security restrictions with the deployment in the baremetal k8s cluster at CESNET we use kaniko instead of Docker to build container images for BinderHub. Thanks to CESNET for the developments in that front.
Please configure the values below accordingly:
- binderhub-secrets.yaml:
token1
andtoken2
must be replaced with a hash generated usingopenssl rand -hex 32
on Linux. Note that the sametoken1
should also be configured in dask-gateway-secrets.yaml. You must also replace the credentials for the container registry (i.e.registry.password
andregistry.username
), and finallydaskbinder.vm.fedcloud.eu
with the DNS hostname of your choice. - binderhub.yaml:
daskbinder.vm.fedcloud.eu
anddaskgateway.vm.fedcloud.eu
must be replaced with your DNS hostname.
Use helm
to install BinderHub with the command below:
sudo helm repo add jupyterhub https://jupyterhub.github.io/helm-chart
sudo helm repo update
sudo helm upgrade binderhub jupyterhub/binderhub \
--install \
--cleanup-on-fail \
--create-namespace \
--namespace binderhub \
--version 1.0.0-0.dev.git.3057.h0a4304c \
--values binderhub/binderhub.yaml \
--values binderhub/binderhub-secrets.yaml
Then deploy Dask Gateway following the steps in the documentation:
sudo helm repo add dask https://helm.dask.org
sudo helm repo update
sudo helm upgrade dask-gateway dask-gateway \
--repo=https://helm.dask.org \
--install \
--cleanup-on-fail \
--create-namespace \
--namespace binderhub \
--version 2023.1.1 \
--values binderhub/dask-gateway.yaml \
--values binderhub/dask-gateway-secrets.yaml
All going well BinderHub will be available at https://daskbinder.vm.fedcloud.eu.
Please configure the values below accordingly:
- minio.yaml:
pangeo-eosc-minio.vm.fedcloud.eu
andpangeo-eosc-minioapi.vm.fedcloud.eu
must be replaced with your DNS hostname. Replacepvc-minio-c-scale
with the corresponding PVC available on your baremetal k8s cluster. - minio-secrets.yaml:
rootUser
androotPassword
must be replaced with your own. Please note that they need to be at least 3 and 8-character long, respectively.
Use helm
to install MinIO with the command below:
helm-v3.11.1/helm upgrade minio bitnami/minio \
--install --wait \
--cleanup-on-fail \
--create-namespace \
--namespace c-scale-pangeo-dask \
--version 14.6.12 \
--values minio/minio.yaml \
--values minio/minio-secrets.yaml
All going well the MinIO Console will be available at https://pangeo-eosc-minio.vm.fedcloud.eu and the MinIO API endpoint will be available at https://pangeo-eosc-minioapi.vm.fedcloud.eu.