Skip to content

Commit d8b6786

Browse files
mrunalpmrunalp
committed
feat(sandbox): add Kubernetes user namespace isolation (hostUsers: false)
Add opt-in support for Kubernetes user namespace isolation on sandbox pods. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced, providing defense-in-depth for the supervisor process. Configuration is two-layered: a cluster-wide default via OPENSHELL_ENABLE_USER_NAMESPACES (default false) and a per-sandbox override via the new `user_namespaces` field on SandboxTemplate. When user namespaces are active, the pod security context is extended with SETUID, SETGID, and DAC_READ_SEARCH capabilities to match the bounding-set requirements inside a user namespace. Introduces SandboxPodParams struct to replace long argument lists on sandbox_to_k8s_spec and sandbox_template_to_k8s. Validated end-to-end on OCP 4.22 (K8s 1.35.3, CRI-O 1.35, RHEL CoreOS, kernel 5.14) with full SSH tunnel and non-identity UID mapping.
1 parent 25c4fde commit d8b6786

14 files changed

Lines changed: 1102 additions & 233 deletions

File tree

‎architecture/kubernetes-user-namespaces-ocp-testing.md‎

Lines changed: 310 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,310 @@
1+
# Testing User Namespaces on OCP
2+
3+
Step-by-step guide to deploy OpenShell with user namespace isolation on an OpenShift cluster and verify end-to-end functionality.
4+
5+
## Prerequisites
6+
7+
- An OCP cluster (tested on OCP 4.22 / K8s 1.35.4 / CRI-O 1.35 / RHEL CoreOS / kernel 5.14)
8+
- `kubectl` and `helm` on your `PATH`
9+
- `podman` for building and pushing images
10+
- `KUBECONFIG` set to point at the cluster
11+
- The OpenShell repo checked out with the user namespace branch built
12+
13+
## 1. Build binaries
14+
15+
```shell
16+
cargo build -p openshell-server --features openshell-core/dev-settings
17+
cargo build -p openshell-sandbox --features openshell-core/dev-settings
18+
cargo build -p openshell-cli --features openshell-core/dev-settings
19+
```
20+
21+
## 2. Create namespace and install the Sandbox CRD
22+
23+
```shell
24+
kubectl create ns openshell
25+
kubectl apply -f deploy/kube/manifests/agent-sandbox.yaml
26+
```
27+
28+
Label the namespace to allow privileged pods:
29+
30+
```shell
31+
kubectl label ns openshell pod-security.kubernetes.io/enforce=privileged --overwrite
32+
kubectl label ns openshell pod-security.kubernetes.io/warn=privileged --overwrite
33+
```
34+
35+
## 3. Grant SCCs
36+
37+
The gateway pod needs `anyuid` (runs as UID 1000) and sandbox pods need `privileged` (capabilities for supervisor):
38+
39+
```shell
40+
kubectl create clusterrolebinding openshell-sa-anyuid \
41+
--clusterrole=system:openshift:scc:anyuid \
42+
--serviceaccount=openshell:openshell
43+
44+
kubectl create clusterrolebinding openshell-sa-privileged \
45+
--clusterrole=system:openshift:scc:privileged \
46+
--serviceaccount=openshell:openshell
47+
48+
kubectl create clusterrolebinding openshell-default-privileged \
49+
--clusterrole=system:openshift:scc:privileged \
50+
--serviceaccount=openshell:default
51+
```
52+
53+
Grant the sandbox CRD controller full permissions (it needs to set ownerReferences with blockOwnerDeletion):
54+
55+
```shell
56+
kubectl create clusterrolebinding agent-sandbox-admin \
57+
--clusterrole=cluster-admin \
58+
--serviceaccount=agent-sandbox-system:agent-sandbox-controller
59+
```
60+
61+
## 4. Generate TLS certificates
62+
63+
```shell
64+
TLSDIR=$(mktemp -d)
65+
66+
# CA
67+
openssl req -x509 -newkey rsa:2048 -nodes \
68+
-keyout $TLSDIR/ca.key -out $TLSDIR/ca.crt \
69+
-days 365 -subj "/CN=openshell-ca" 2>/dev/null
70+
71+
# Server cert
72+
openssl req -newkey rsa:2048 -nodes \
73+
-keyout $TLSDIR/server.key -out $TLSDIR/server.csr \
74+
-subj "/CN=openshell.openshell.svc.cluster.local" \
75+
-addext "subjectAltName=DNS:openshell.openshell.svc.cluster.local,DNS:openshell,DNS:localhost,IP:127.0.0.1" 2>/dev/null
76+
77+
openssl x509 -req -in $TLSDIR/server.csr \
78+
-CA $TLSDIR/ca.crt -CAkey $TLSDIR/ca.key -CAcreateserial \
79+
-out $TLSDIR/server.crt -days 365 \
80+
-extfile <(echo "subjectAltName=DNS:openshell.openshell.svc.cluster.local,DNS:openshell,DNS:localhost,IP:127.0.0.1") 2>/dev/null
81+
82+
# Client cert
83+
openssl req -newkey rsa:2048 -nodes \
84+
-keyout $TLSDIR/client.key -out $TLSDIR/client.csr \
85+
-subj "/CN=openshell-client" 2>/dev/null
86+
87+
openssl x509 -req -in $TLSDIR/client.csr \
88+
-CA $TLSDIR/ca.crt -CAkey $TLSDIR/ca.key -CAcreateserial \
89+
-out $TLSDIR/client.crt -days 365 2>/dev/null
90+
```
91+
92+
Create Kubernetes secrets:
93+
94+
```shell
95+
kubectl create secret tls openshell-server-tls -n openshell \
96+
--cert=$TLSDIR/server.crt --key=$TLSDIR/server.key
97+
98+
kubectl create secret generic openshell-server-client-ca -n openshell \
99+
--from-file=ca.crt=$TLSDIR/ca.crt
100+
101+
kubectl create secret generic openshell-client-tls -n openshell \
102+
--from-file=ca.crt=$TLSDIR/ca.crt \
103+
--from-file=tls.crt=$TLSDIR/client.crt \
104+
--from-file=tls.key=$TLSDIR/client.key
105+
106+
kubectl create secret generic openshell-ssh-handshake -n openshell \
107+
--from-literal=secret=$(openssl rand -hex 32)
108+
```
109+
110+
Note: the `openshell-client-tls` secret must include `ca.crt`, `tls.crt`, and `tls.key` (not a `kubernetes.io/tls` type secret, which only has `tls.crt` and `tls.key`).
111+
112+
## 5. Expose the OCP internal registry and push images
113+
114+
```shell
115+
# Enable the default route for the internal registry
116+
kubectl patch configs.imageregistry.operator.openshift.io/cluster \
117+
--type merge -p '{"spec":{"defaultRoute":true}}'
118+
119+
sleep 5
120+
REGISTRY=$(kubectl get route default-route -n openshift-image-registry -o jsonpath='{.spec.host}')
121+
TOKEN=$(kubectl create token builder -n openshell)
122+
123+
podman login --tls-verify=false -u kubeadmin -p "$TOKEN" "$REGISTRY"
124+
```
125+
126+
Build and push the gateway image. The Dockerfile expects prebuilt binaries, so stage the locally-built binary first:
127+
128+
```shell
129+
mkdir -p deploy/docker/.build/prebuilt-binaries/amd64
130+
cp target/debug/openshell-gateway deploy/docker/.build/prebuilt-binaries/amd64/
131+
132+
podman build -f deploy/docker/Dockerfile.images --target gateway \
133+
-t localhost/openshell/gateway:dev .
134+
135+
rm -rf deploy/docker/.build/prebuilt-binaries
136+
137+
podman push --tls-verify=false localhost/openshell/gateway:dev \
138+
$REGISTRY/openshell/gateway:dev
139+
```
140+
141+
Pull and push the sandbox base image:
142+
143+
```shell
144+
podman pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest
145+
146+
podman push --tls-verify=false \
147+
ghcr.io/nvidia/openshell-community/sandboxes/base:latest \
148+
$REGISTRY/openshell/sandbox-base:latest
149+
```
150+
151+
## 6. Build and push the supervisor image
152+
153+
The sandbox supervisor binary is distributed to pods via an init container that copies it from a container image. The binary must be at `/usr/local/bin/openshell-sandbox` so the init container can find it via `command -v`.
154+
155+
```shell
156+
cp target/debug/openshell-sandbox /tmp/openshell-sandbox
157+
158+
cat > /tmp/Dockerfile.supervisor <<'EOF'
159+
FROM registry.access.redhat.com/ubi9/ubi-minimal:latest
160+
COPY openshell-sandbox /usr/local/bin/openshell-sandbox
161+
RUN chmod 755 /usr/local/bin/openshell-sandbox
162+
EOF
163+
164+
podman build -f /tmp/Dockerfile.supervisor \
165+
-t localhost/openshell/supervisor:latest /tmp/
166+
167+
podman push --tls-verify=false localhost/openshell/supervisor:latest \
168+
$REGISTRY/openshell/supervisor:latest
169+
```
170+
171+
## 7. Deploy the gateway with Helm
172+
173+
```shell
174+
INTERNAL_REG="image-registry.openshift-image-registry.svc:5000"
175+
176+
helm install openshell deploy/helm/openshell -n openshell \
177+
--set image.repository=$INTERNAL_REG/openshell/gateway \
178+
--set image.tag=dev \
179+
--set image.pullPolicy=Always \
180+
--set server.sandboxImage="$INTERNAL_REG/openshell/sandbox-base:latest" \
181+
--set server.sandboxImagePullPolicy=Always \
182+
--set server.supervisorImage="$INTERNAL_REG/openshell/supervisor:latest" \
183+
--set server.supervisorImagePullPolicy=Always \
184+
--set server.enableUserNamespaces=true \
185+
--set server.grpcEndpoint="https://openshell.openshell.svc.cluster.local:8080" \
186+
--set server.dbUrl="sqlite:/tmp/openshell.db" \
187+
--set service.type=ClusterIP
188+
```
189+
190+
Wait for the gateway to be ready:
191+
192+
```shell
193+
kubectl rollout status statefulset/openshell -n openshell --timeout=120s
194+
```
195+
196+
Note: `server.dbUrl` is set to `/tmp/openshell.db` to avoid PVC permission issues on clusters without a properly configured storage class. For production, use a PVC-backed path.
197+
198+
## 8. Configure the CLI
199+
200+
Port-forward the gateway service to localhost:
201+
202+
```shell
203+
nohup kubectl port-forward svc/openshell -n openshell 18443:8080 >/tmp/pf.log 2>&1 &
204+
```
205+
206+
Set up the CLI gateway configuration with mTLS:
207+
208+
```shell
209+
mkdir -p ~/.config/openshell/gateways/ocp-userns/mtls
210+
211+
cp $TLSDIR/ca.crt ~/.config/openshell/gateways/ocp-userns/mtls/
212+
cp $TLSDIR/client.crt ~/.config/openshell/gateways/ocp-userns/mtls/tls.crt
213+
cp $TLSDIR/client.key ~/.config/openshell/gateways/ocp-userns/mtls/tls.key
214+
215+
cat > ~/.config/openshell/gateways/ocp-userns/metadata.json <<'EOF'
216+
{
217+
"name": "ocp-userns",
218+
"gateway_endpoint": "https://127.0.0.1:18443",
219+
"is_remote": false,
220+
"gateway_port": 18443,
221+
"auth_mode": "mtls"
222+
}
223+
EOF
224+
```
225+
226+
Verify connectivity:
227+
228+
```shell
229+
OPENSHELL_GATEWAY=ocp-userns target/debug/openshell status
230+
```
231+
232+
Expected output:
233+
234+
```
235+
Server Status
236+
Gateway: ocp-userns
237+
Server: https://127.0.0.1:18443
238+
Status: Connected
239+
```
240+
241+
## 9. Create a sandbox and verify user namespaces
242+
243+
```shell
244+
export OPENSHELL_GATEWAY=ocp-userns
245+
246+
target/debug/openshell sandbox create --no-bootstrap -- sh -lc \
247+
"echo '=== uid_map ==='; cat /proc/self/uid_map; \
248+
echo '=== gid_map ==='; cat /proc/self/gid_map; \
249+
echo '=== id ==='; id; \
250+
echo '=== userns-e2e-ok ==='"
251+
```
252+
253+
Expected output (UID values will vary):
254+
255+
```
256+
=== uid_map ===
257+
0 4176084992 65536
258+
=== gid_map ===
259+
0 4176084992 65536
260+
=== id ===
261+
uid=998(sandbox) gid=998(sandbox) groups=998(sandbox)
262+
=== userns-e2e-ok ===
263+
```
264+
265+
This confirms:
266+
267+
- UID 0 inside the container maps to a high host UID (non-identity mapping)
268+
- The sandbox user (UID 998) is active
269+
- The SSH tunnel through the gateway works end-to-end
270+
- Workspace init, supervisor startup, network namespace creation, and proxy all function correctly under user namespace isolation
271+
272+
## 10. Cleanup
273+
274+
```shell
275+
# Delete all sandboxes
276+
kubectl delete sandbox --all -n openshell
277+
278+
# Uninstall the Helm release
279+
helm uninstall openshell -n openshell
280+
281+
# Remove RBAC
282+
kubectl delete clusterrolebinding openshell-sa-anyuid openshell-sa-privileged \
283+
openshell-default-privileged agent-sandbox-admin 2>/dev/null
284+
285+
# Remove the Sandbox CRD and its controller
286+
kubectl delete -f deploy/kube/manifests/agent-sandbox.yaml
287+
288+
# Remove the namespace
289+
kubectl delete ns openshell
290+
291+
# Kill port-forward
292+
pkill -f "port-forward.*18443"
293+
294+
# Remove CLI gateway config
295+
rm -rf ~/.config/openshell/gateways/ocp-userns
296+
```
297+
298+
## Troubleshooting
299+
300+
| Symptom | Cause | Fix |
301+
|---------|-------|-----|
302+
| `ErrImageNeverPull` on gateway pod | Image not in the internal registry | Push with `podman push --tls-verify=false` to the OCP registry |
303+
| `unable to validate against any security context constraint` | Missing SCC grants | Run the `clusterrolebinding` commands from step 3 |
304+
| `cannot set blockOwnerDeletion` on sandbox creation | Sandbox CRD controller lacks RBAC | Grant `cluster-admin` to the controller SA (step 3) |
305+
| Supervisor init container `CrashLoopBackOff` with empty logs | Supervisor binary not in `$PATH` inside image | Ensure the binary is at `/usr/local/bin/openshell-sandbox` in the supervisor image (step 6) |
306+
| `mount of /sys failed: Permission denied` in sandbox | Published supervisor image used instead of local build | Set `server.supervisorImage` in Helm to your internal registry image (step 7) |
307+
| `failed to set MOUNT_ATTR_IDMAP` | Filesystem doesn't support ID-mapped mounts | Only happens in nested container environments (DinD); native nodes work |
308+
| Gateway pod `CrashLoopBackOff` with `unable to open database file` | PVC permissions | Use `--set server.dbUrl="sqlite:/tmp/openshell.db"` |
309+
| Gateway pod `CrashLoopBackOff` with `unexpected argument '--bind-address'` | Stale gateway image from build cache | Rebuild with `--no-cache` or stage fresh binary at `deploy/docker/.build/prebuilt-binaries/amd64/` |
310+
| `dns error: failed to lookup address` from supervisor | In-cluster DNS not resolving | Use the ClusterIP directly in `server.grpcEndpoint` instead of the DNS name |

0 commit comments

Comments
 (0)