SciDX Kubernetes Resources

Deployable configs for the SciDX services stack on Kubernetes: JupyterHub, CKAN, Kafka, and the NDP Endpoint API plus admin console. Core components live at the repo root; optional helpers are put under contrib/.

Components at a glance

Component	Location	Notes
JupyterHub	ndp-jupyterhub/ (submodule)	Helm config and docs with Makefile
CKAN	ckan-helm/ (submodule)	Helm chart fork with Makefile
Kafka	kafka/	Strimzi provisioned Kafka cluster with Makefile
NDP Endpoint API	ep-api-kustomize/	Kustomize overlays
NDP Endpoint Admin Console	ep-frontend-kustomize/	Kustomize overlays
Optional helpers	contrib/	non-core helpers

Prerequisites

• kubectl 1.21+ against a cluster/context where you can create namespaces, ingresses, and PVCs.
• Helm 3, kustomize support (built into recent kubectl), and make.
• Git with SSH access to GitHub if you use the git@ submodule URLs; switch .gitmodules to HTTPS if needed.

Clone this repo

Submodule Caveat: ndp-jupyterhub is currently a private repo. Make sure you can access https://github.com/national-data-platform/ndp-jupyterhub.git or the submodule checkout will fail during clone, consult NDP admin if you need access.

git clone --recurse-submodules https://github.com/sci-ndp/scidx-k8s.git

cd scidx-k8s

If you already cloned without submodules git submodule update --init --recursive

Installation

1. Confirm cluster context availability:

   kubectl config current-context

   Optional: install ingress-nginx controller if your cluster lacks an ingress controller, see contrib/nginx-ingress-controller

2. Deploy Kafka (Strimzi):

   cd kafka

   and follow Kafka Deployment Document.

3. Deploy CKAN:

   cd ..  # go back to repo root
   cd ckan-helm

   and follow CKAN Deployment Document.

4. Deploy JupyterHub:

   cd ..  # go back to repo root
   cd ndp-jupyterhub/helm-generic

   and follow generic NDP JupyterHub Deployment Document.

5. Deploy NDP Endpoint API:

   cd ../..  # go back to repo root
   cd ep-api-kustomize

   and follow NDP Endpoint API Deployment Document.

6. Deploy NDP Endpoint Admin Console:

   cd ..  # go back to repo root
   cd ep-frontend-kustomize

   and follow NDP Endpoint Admin Console Deployment Document.

Deployed services (what you should have running)

Service	Default namespace	Access	What to expect	Quick check
Kafka (Strimzi)	kafka	Service + TCP passthrough	Strimzi operator + Kafka broker pods	kubectl get pods,svc -n kafka
CKAN	ckan	Ingress (/ckan)	Web UI, API, and worker pods	kubectl get pods,svc,ingress -n ckan
NDP Generic JupyterHub	ndp-jhub	Ingress (/jupyter/)	Hub pods + proxy	kubectl get pods,svc,ingress -n ndp-jhub
NDP Endpoint API	ndp-ep, ndp-ep-dev, ndp-ep-test	Ingress(/api for prod)	API deployment + service + ingress	kubectl get pods,svc,ingress -n <overlay-namespace>
NDP Endpoint Admin Console	ep-frontend, ep-frontend-dev, ndp-ep-console-test	Ingress	Frontend deployment + service + ingress	kubectl get pods,svc,ingress -n <overlay-namespace>

Note: defaults are from each component's Helm/kustomize config; update the namespaces if you changed them.

Optional helpers under contrib/ are not part of the core stack and used when needed.