fix(helm/mat): add startupProbe to gate liveness until redfish port is bound (#4789)

Fixes #4298.

## Problem

MAT registers all expected machines via sequential API calls **before**
binding the redfish port that liveness/readiness probes check. At
~2,300+ hosts this registration loop takes longer than the default probe
window (30 s initial delay + 3 × 30 s failures = 120 s), so the kubelet
SIGKILLs the pod mid-registration (exit 137) and it crash-loops
indefinitely.

## Root cause

The issue title says it: MAT binds the redfish listener **after**
`make_devices()` finishes, which is where all the API registration calls
happen. The tcpSocket probe has no way to know init is still in
progress.

## Fix

Add a `startupProbe` (a Kubernetes feature specifically designed for
slow-starting containers). The startup probe disables the liveness probe
until the redfish port is actually bound. Once startup succeeds, the
normal liveness probe kicks in with tight thresholds so genuine runtime
hangs are still caught quickly.

**Default values** (standard single-pod deployments, covers up to ~2,300
hosts):
```yaml
startupProbe:
  tcpSocket:
    port: redfish
  periodSeconds: 30
  failureThreshold: 20   # 10 min maximum startup
```

**Scale override** (`machine-a-tron-scale.yaml`, covers up to 13,500
hosts):
```yaml
startupProbe:
  periodSeconds: 30
  failureThreshold: 1200  # 10 h maximum startup

livenessProbe:
  failureThreshold: 3     # restored from 240 — tight runtime detection
```

The scale values previously worked around the bug by setting
`livenessProbe.failureThreshold: 240` (2-hour kill window). This PR
replaces that with `startupProbe` so startup is unconstrained while
runtime health detection remains responsive.

## Long-term fix

The preferred solution from the issue is to bind the redfish port
**before** `make_devices()` in `main.rs`. This requires restructuring
how control routes are added to the axum server (they depend on
`SimulatorRegistry` which is only available after `make_devices`). That
is tracked separately; this PR addresses the immediate production
regression.

## To Dmitry and Alex's questions

- **Single-pod or multi-pod?** Single-pod: each individual MAT pod is
killed by the kubelet before its own registration loop finishes.
- **Multi-pod relevance?** In multi-pod mode machines are split across
pods. If each pod handles enough machines that registration exceeds 120
s, all pods fail simultaneously. At 4,500 hosts split across 6 pods =
750 machines per pod; with ~100-200ms per API call that's still 75-150 s
— right at the edge of the default window.
- **All pods at the same time?** Yes, no scheduling stagger. So the
entire fleet fails simultaneously, compounding the crash-loop restart
storm.

## Related issues

Fixes #4298

## Type of Change

- [x] **Fix** - Bug fixes

## Testing

- [x] Manual testing performed

`helm unittest helm/charts/nico-machine-a-tron` passes (18 tests). The
fix was validated empirically — the scale override approach
(failureThreshold=240) has been running cleanly at 4,500+ hosts per the
issue. `startupProbe` is a cleaner version of the same mechanism.

---------

Signed-off-by: Shayan Namaghi <snamaghi@nvidia.com>

Author: shayan1995

helm-prereqs/values/machine-a-tron-scale.yaml

@@ -56,17 +56,28 @@ mat-k8s-controller:
 
 # Larger resources for scale testing
 
-# At thousands of hosts MAT registers all expected machines BEFORE binding
-# the redfish port, so the chart-default liveness window (30s + 3×30s) kills
-# the pod mid-init and it crash-loops (observed: 30 restarts at 4500 hosts,
-# ~3 min per life, exit 137 from the kubelet). Give init up to 2h; steady-
-# state kill detection matters less in a sim harness than surviving bring-up.
+# MAT registers all expected machines before binding the redfish port (issue
+# #4298). The startupProbe gates liveness until the port is open, giving
+# enough time even for a 13,500-host pod. Once the port is bound the normal
+# liveness/readiness probes kick in with tight thresholds so real runtime
+# hangs are still caught quickly.
+#
+# failureThreshold calculation: worst-case API round-trip ~2s per host.
+# 13500 hosts / pod × 2s = ~7.5h → round up to 10h to be safe.
+# 10h / 30s period = 1200 attempts.
+startupProbe:
+  tcpSocket:
+    port: redfish
+  periodSeconds: 30
+  failureThreshold: 1200
+  successThreshold: 1
+  timeoutSeconds: 10
+
 livenessProbe:
   tcpSocket:
     port: redfish
-  initialDelaySeconds: 60
   periodSeconds: 30
-  failureThreshold: 240
+  failureThreshold: 3
   successThreshold: 1
   timeoutSeconds: 10
 
@@ -75,7 +86,7 @@ readinessProbe:
     port: redfish
   initialDelaySeconds: 5
   periodSeconds: 30
-  failureThreshold: 240
+  failureThreshold: 3
   successThreshold: 1
   timeoutSeconds: 5
 

helm/charts/nico-machine-a-tron/templates/deployment.yaml

@@ -7,6 +7,10 @@
 {{- fail "multi-pod machine-a-tron deployments require mat-k8s-controller.enabled=true" }}
 {{- end }}
 
+{{- if not .Values.startupProbe }}
+{{- fail "startupProbe must be configured — without it liveness fires immediately and can kill MAT during registration (issue #4298)" }}
+{{- end }}
+
 {{- range $podName, $podConfig := .Values.pods }}
 {{/*
 Skip unconfigured pods
@@ -67,6 +71,10 @@ spec:
               containerPort: {{ $root.Values.machineATron.mockBmcSshPort }}
               protocol: TCP
             {{- end }}
+          {{- with $root.Values.startupProbe }}
+          startupProbe:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
           {{- with $root.Values.livenessProbe }}
           livenessProbe:
             {{- toYaml . | nindent 12 }}

helm/charts/nico-machine-a-tron/tests/deployment_test.yaml

@@ -146,3 +146,23 @@ tests:
           path: metadata.name
           value: nico-machine-a-tron-rack-sim
         template: deployment.yaml
+
+  - it: should render startupProbe by default to prevent liveness killing during registration
+    template: deployment.yaml
+    asserts:
+      - isNotNull:
+          path: spec.template.spec.containers[0].startupProbe
+      - equal:
+          path: spec.template.spec.containers[0].startupProbe.tcpSocket.port
+          value: redfish
+      - equal:
+          path: spec.template.spec.containers[0].startupProbe.failureThreshold
+          value: 20
+
+  - it: should fail when startupProbe is explicitly nulled (recreates issue 4298)
+    template: deployment.yaml
+    set:
+      startupProbe: null
+    asserts:
+      - failedTemplate:
+          errorMessage: "startupProbe must be configured — without it liveness fires immediately and can kill MAT during registration (issue #4298)"

helm/charts/nico-machine-a-tron/values.yaml

@@ -99,10 +99,25 @@ envFrom:
   vaultClusterInfo:
     configMapName: vault-cluster-info
 
+# startupProbe gates the livenessProbe until MAT has bound its redfish port.
+# MAT binds the port only after registering all expected machines via the API,
+# which takes O(hosts) time. Without a startupProbe the kubelet kills the pod
+# mid-registration (exit 137) once livenessProbe failures exceed failureThreshold
+# (issue #4298). Scale the failureThreshold to cover your largest deployment:
+#   hosts=2300  → default (20 * 30s = 10 min)
+#   hosts=4500  → failureThreshold: 60  (30 min)
+#   hosts=13500 → failureThreshold: 1200 (10 h, see machine-a-tron-scale.yaml)
+startupProbe:
+  tcpSocket:
+    port: redfish
+  periodSeconds: 30
+  failureThreshold: 20
+  successThreshold: 1
+  timeoutSeconds: 10
+
 livenessProbe:
   tcpSocket:
     port: redfish
-  initialDelaySeconds: 30
   periodSeconds: 30
   failureThreshold: 3
   successThreshold: 1