feat: add kubernetes local-dev environment

* Add support for grpcRoute from Kubernetes Gateway API spec
* Add pkiInitJob to initialize mTLS resources
* Add sshHandshake init job
* Test integration with Envoy Gateway
* Add keycloak integration testing with Skaffold

Author: TaylorMutch

.agents/skills/helm-dev-environment/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: helm-dev-environment
+description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local.
+---
+
+# Helm Dev Environment
+
+Set up, run, and tear down the local Kubernetes development environment for OpenShell.
+The stack is: **k3d** (Docker-backed k3s) for the cluster, **Skaffold** for image builds and Helm deploys, and the **OpenShell Helm chart** (`deploy/helm/openshell/`).
+
+---
+
+## Prerequisites
+
+- Docker Desktop (macOS) or Docker Engine (Linux) running
+- `mise install` completed (provides `k3d`, `kubectl`, `skaffold`, `helm`)
+
+---
+
+## Startup
+
+### 1. Create the cluster
+
+```bash
+mise run helm:k3s:create
+```
+
+Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file.
+Also applies base manifests (`deploy/kube/manifests/agent-sandbox.yaml`). Traefik is
+disabled at cluster creation time.
+
+**Multi-worktree support:** the cluster name is derived from the last component of the
+current git branch (e.g. branch `kube-support/local-dev/tmutch` → cluster
+`openshell-dev-tmutch`). Each worktree therefore gets its own isolated cluster and its
+own `kubeconfig` file. Override with `HELM_K3S_CLUSTER_NAME` to force a specific name
+or share one cluster across worktrees.
+
+Port mappings created at cluster time (cannot be changed without recreating):
+
+| Host port | Target | Used by |
+|-----------|--------|---------|
+| `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) |
+
+Override with env vars before running `helm:k3s:create`:
+- `HELM_K3S_LB_HOST_PORT` (default: `8080`)
+
+### 2. Deploy OpenShell
+
+**Iterative dev** (rebuilds on file changes, recommended during active development):
+```bash
+mise run helm:skaffold:dev
+```
+
+**One-shot deploy** (build once and leave running):
+```bash
+mise run helm:skaffold:run
+```
+
+Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
+chart. The `pkiInitJob` hook runs on first install to generate mTLS secrets. Envoy Gateway is opt-in; see the Optional Add-ons section below.
+
+The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`.
+
+---
+
+## Teardown
+
+### Remove the Helm releases (keep cluster)
+
+```bash
+mise run helm:skaffold:delete
+```
+
+### Delete the cluster entirely
+
+```bash
+mise run helm:k3s:delete
+```
+
+This removes the k3d cluster and all resources. Kubeconfig context is left behind
+but will point to a deleted cluster — safe to ignore or clean up manually.
+
+---
+
+## Optional Add-ons
+
+Each add-on requires uncommenting the corresponding `valuesFiles` entry in
+`deploy/helm/openshell/skaffold.yaml` before running `helm:skaffold:dev` or `helm:skaffold:run`.
+
+### Envoy Gateway (Gateway API / GRPCRoute)
+
+Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release in
+`skaffold.yaml`). To activate routing:
+
+1. Uncomment `#- values-gateway.yaml` in `skaffold.yaml`
+2. Redeploy: `mise run helm:skaffold:run`
+3. Apply the GatewayClass: `mise run helm:gateway:apply`
+4. Access: `http://127.0.0.1:8080`
+
+`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and a
+`GRPCRoute` in the `openshell` namespace. Envoy Gateway provisions a LoadBalancer
+service for the proxy; klipper-lb binds it to hostPort 80, reachable via the
+`8080:80` load balancer port mapping.
+
+### Keycloak OIDC
+
+One-time setup — only needed once per cluster lifetime:
+
+```bash
+mise run keycloak:k8s:setup
+```
+
+This deploys Keycloak (`quay.io/keycloak/keycloak:24.0`) into the `keycloak` namespace,
+imports the openshell realm from `scripts/keycloak-realm.json`, and prints a port-forward
+command for acquiring tokens from the CLI.
+
+Then activate OIDC in the OpenShell Helm chart:
+1. Uncomment `#- values-keycloak.yaml` in `skaffold.yaml`
+2. Redeploy: `mise run helm:skaffold:run`
+
+To remove Keycloak:
+```bash
+mise run keycloak:k8s:teardown
+```
+
+---
+
+## Cluster Lifecycle (suspend/resume)
+
+Stop the cluster without losing state (faster than delete/recreate):
+```bash
+mise run helm:k3s:stop
+mise run helm:k3s:start
+```
+
+Check cluster status:
+```bash
+mise run helm:k3s:status
+```
+
+---
+
+## Key Files
+
+| Path | Purpose |
+|------|---------|
+| `deploy/helm/openshell/skaffold.yaml` | Skaffold config — images, Helm releases, values overlays |
+| `deploy/helm/openshell/values.yaml` | Default Helm values |
+| `deploy/helm/openshell/values-skaffold.yaml` | Dev overrides (image pull policy, local image names) |
+| `deploy/helm/openshell/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
+| `deploy/helm/openshell/values-keycloak.yaml` | Keycloak OIDC overlay |
+| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) |
+| `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
+| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy + realm import |

deploy/docker/Dockerfile.images

@@ -13,6 +13,10 @@
 #
 # Rust binaries are built natively before the image build and staged at:
 #   deploy/docker/.build/prebuilt-binaries/<arch>/openshell-{gateway,sandbox}
+#
+# For local dev (Skaffold), pass --build-arg BUILD_FROM_SOURCE=1 to compile
+# binaries inside Docker instead. BuildKit only executes the selected binary
+# staging stage, so missing prebuilt files do not cause a build failure.
 
 # Pin by tag AND manifest-list digest to prevent silent upstream republishes
 # from breaking the build. Update both when bumping k3s versions.
@@ -22,22 +26,67 @@ ARG K3S_DIGEST=sha256:4607083d3cac07e1ccde7317297271d13ed5f60f35a78f33fcef84858a
 ARG K9S_VERSION=v0.50.18
 ARG HELM_VERSION=v3.17.3
 ARG NVIDIA_CONTAINER_TOOLKIT_VERSION=1.18.2-1
+# Controls binary source: 0 = prebuilt (release), 1 = compile in Docker (local dev).
+# Must be declared here (global scope) so it can be used in FROM instructions below.
+ARG BUILD_FROM_SOURCE=0
+
+# ---------------------------------------------------------------------------
+# Optional in-Docker Rust build (BUILD_FROM_SOURCE=1, local dev only)
+# ---------------------------------------------------------------------------
+FROM rust:1.95.0-slim-bookworm AS rust-builder
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    cmake \
+    pkg-config \
+    libssl-dev \
+    ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /build
+
+COPY Cargo.toml Cargo.lock ./
+COPY crates/ crates/
+COPY proto/ proto/
+
+RUN --mount=type=cache,target=/usr/local/cargo/registry \
+    --mount=type=cache,target=/build/target \
+    cargo build --release \
+        --features "openshell-core/dev-settings" \
+        --bin openshell-gateway \
+        --bin openshell-sandbox \
+    && mkdir -p /build/out \
+    && install -m 0755 target/release/openshell-gateway /build/out/openshell-gateway \
+    && install -m 0755 target/release/openshell-sandbox /build/out/openshell-sandbox
 
 # ---------------------------------------------------------------------------
 # Per-arch binary stages
 # ---------------------------------------------------------------------------
-FROM scratch AS gateway-binary
+
+# Prebuilt path (release default, BUILD_FROM_SOURCE=0)
+FROM scratch AS gateway-binary-0
 ARG TARGETARCH
 # --chmod=755 preserves the executable bit through actions/upload-artifact +
 # download-artifact, which strip exec perms during the roundtrip.
 COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-gateway /build/out/openshell-gateway
 
-FROM scratch AS supervisor-binary
+# Source-built path (local dev, BUILD_FROM_SOURCE=1)
+FROM rust-builder AS gateway-binary-1
+
+FROM gateway-binary-${BUILD_FROM_SOURCE} AS gateway-binary
+
+# Prebuilt path (release default, BUILD_FROM_SOURCE=0)
+FROM scratch AS supervisor-binary-0
 ARG TARGETARCH
 # --chmod=755 preserves the executable bit through actions/upload-artifact +
 # download-artifact, which strip exec perms during the roundtrip.
 COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /build/out/openshell-sandbox
 
+# Source-built path (local dev, BUILD_FROM_SOURCE=1)
+FROM rust-builder AS supervisor-binary-1
+
+FROM supervisor-binary-${BUILD_FROM_SOURCE} AS supervisor-binary
+
 # Minimal extraction stage for fast-deploy: exports only the supervisor
 # binary (~20-40 MB) instead of the entire build environment (~968 MB).
 FROM scratch AS supervisor-output

deploy/helm/openshell/.helmignore

@@ -16,3 +16,10 @@
 .idea/
 *.tmproj
 .vscode/
+
+# Ignore development files
+skaffold.yaml
+values-keycloak.yaml
+values-ingress.yaml
+values-gateway.yaml
+values-skaffold.yaml

deploy/helm/openshell/Chart.yaml

@@ -5,5 +5,8 @@ apiVersion: v2
 name: openshell
 description: runtime environment for autonomous agents
 type: application
-version: 0.1.0
-appVersion: "0.1.0"
+# Updated to the release version by CI. The appVersion doubles as the default
+# image tag (image.tag defaults to appVersion when empty), so a released chart
+# automatically pulls the matching gateway and supervisor images.
+version: 0.0.0
+appVersion: "0.0.0"

deploy/helm/openshell/skaffold.yaml

@@ -0,0 +1,90 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Local dev: builds gateway + supervisor images using Dockerfile.images with
+# BUILD_FROM_SOURCE=1, which compiles Rust binaries inside Docker without
+# requiring pre-staged artifacts.
+#
+# Run from repo root:
+#   skaffold dev -f deploy/helm/openshell/skaffold.yaml
+#
+# See https://skaffold.dev/docs/deployers/helm/ (setValueTemplates, IMAGE_* fields).
+apiVersion: skaffold/v4beta14
+kind: Config
+metadata:
+  name: openshell
+build:
+  local:
+    push: false
+  tagPolicy:
+    gitCommit: {}
+  artifacts:
+    - image: openshell/gateway
+      context: ../../..
+      custom:
+        buildCommand: |
+          docker buildx build \
+            --build-arg BUILD_FROM_SOURCE=1 \
+            --target gateway \
+            --tag "$IMAGE" \
+            --load \
+            --file deploy/docker/Dockerfile.images \
+            .
+        dependencies:
+          paths:
+            - Cargo.toml
+            - Cargo.lock
+            - crates/**
+            - proto/**
+            - deploy/docker/Dockerfile.images
+            - crates/openshell-server/migrations/**
+    - image: openshell/supervisor
+      context: ../../..
+      custom:
+        buildCommand: |
+          docker buildx build \
+            --build-arg BUILD_FROM_SOURCE=1 \
+            --target supervisor \
+            --tag "$IMAGE" \
+            --load \
+            --file deploy/docker/Dockerfile.images \
+            .
+        dependencies:
+          paths:
+            - Cargo.toml
+            - Cargo.lock
+            - crates/**
+            - proto/**
+            - deploy/docker/Dockerfile.images
+deploy:
+  helm:
+    releases:
+      # Envoy Gateway — Kubernetes Gateway API implementation.
+      # Installs the Gateway API CRDs and the "eg" GatewayClass.
+      # Required when grpcRoute.enabled is true in the openshell release.
+      #- name: envoy-gateway
+      #  remoteChart: oci://docker.io/envoyproxy/gateway-helm
+      #  version: v1.7.2
+      #  namespace: envoy-gateway-system
+      #  createNamespace: true
+      #  # wait ensures Gateway API CRDs are registered before the openshell
+      #  # release attempts to create Gateway and HTTPRoute resources.
+      #  wait: true
+      - name: openshell
+        chartPath: .
+        namespace: openshell
+        createNamespace: true
+        valuesFiles:
+          - values.yaml
+          - values-skaffold.yaml
+          # To enable OIDC with a local Keycloak instance, run the one-time
+          # setup task first, then uncomment the line below:
+          #   mise run keycloak:k8s:setup
+          #- values-keycloak.yaml
+          # To enable the Gateway API HTTPRoute (requires Envoy Gateway above):
+          #- values-gateway.yaml
+        setValueTemplates:
+          image.repository: '{{.IMAGE_REPO_openshell_gateway}}'
+          image.tag: '{{.IMAGE_TAG_openshell_gateway}}'
+          supervisor.image.repository: '{{.IMAGE_REPO_openshell_supervisor}}'
+          supervisor.image.tag: '{{.IMAGE_TAG_openshell_supervisor}}'

deploy/helm/openshell/templates/_helpers.tpl

@@ -58,3 +58,26 @@ Create the name of the service account to use
 {{- default "default" .Values.serviceAccount.name }}
 {{- end }}
 {{- end }}
+
+{{/*
+Gateway image reference. Uses image.tag when set; falls back to .Chart.AppVersion
+so a released chart automatically pulls the matching image without extra overrides.
+*/}}
+{{- define "openshell.image" -}}
+{{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) }}
+{{- end }}
+
+{{/*
+Supervisor image reference. Same appVersion fallback as openshell.image so
+the supervisor and gateway images stay in sync across releases.
+*/}}
+{{- define "openshell.supervisorImage" -}}
+{{- printf "%s:%s" .Values.supervisor.image.repository (.Values.supervisor.image.tag | default .Chart.AppVersion) }}
+{{- end }}
+
+{{/*
+Namespaced Issuer (selfSigned) for cert-manager CA bootstrap.
+*/}}
+{{- define "openshell.issuerSelfSigned" -}}
+{{- printf "%s-selfsigned" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }}
+{{- end }}

deploy/helm/openshell/templates/gateway.yaml

@@ -0,0 +1,21 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.grpcRoute.enabled .Values.grpcRoute.gateway.create }}
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: {{ default (include "openshell.fullname" .) .Values.grpcRoute.gateway.name }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "openshell.labels" . | nindent 4 }}
+spec:
+  gatewayClassName: {{ .Values.grpcRoute.gateway.className }}
+  listeners:
+    - name: http
+      port: {{ .Values.grpcRoute.gateway.listener.port }}
+      protocol: {{ .Values.grpcRoute.gateway.listener.protocol }}
+      allowedRoutes:
+        namespaces:
+          from: {{ .Values.grpcRoute.gateway.listener.allowedRoutes }}
+{{- end }}