fix(helm-prereqs): size the postgres default for the fleet, and warn when it is too small

The default was 4Gi, and at 4,500 hosts that is not survivable. Postgres running
out of memory does not degrade gracefully: the kernel OOM-kills a backend, the
postmaster crash-recovers, and every open connection is dropped with in-flight
transactions rolled back.

Measured on a 4,500-host (13,500 machine) ingestion:

  4Gi  -> 99.96% memory, 531 OOM kills, 112 crash-recoveries, never completed
  24Gi -> ~8.7GiB peak, zero OOM kills, full fleet ingested with every machine
          reaching ready (reproduced across three runs at 54-56 machines/min)

The node had 240GB free, so this was a configured limit rather than a hardware
constraint.

What made it expensive is that the failure presents downstream: three separate
issues were filed as product defects during the period it was in effect
(readiness plateau #4750, cleanup panic #4753, and a Postgres deadlock) and none
reproduced once the memory was adequate. Days went into tracing individual state
machines before the memory pressure was noticed.

Raise the default to 16Gi/8CPU (production is documented as 32 CPU / 16Gi, so
this aligns rather than exceeds it) and replace the 'tune down for dev clusters'
comment -- which reads as though 4Gi is a safe dev value -- with fleet-size
guidance and the one-line cgroup check that identifies this condition.

Also warn in setup-machine-a-tron.sh Phase 0 when the live limit is below what
the requested host count wants, so an undersized database is visible at minute
zero instead of hour three. The warning names the reinstall requirement too:
teardown deletes the postgres namespace, so a resize applied to a live cluster
before a reinstall is silently reverted.

Fixes #4839

Author: shayan1995

helm-prereqs/setup-machine-a-tron.sh

@@ -358,6 +358,35 @@ for _pw in "$BMC_PASSWORD" "$UEFI_DPU_PASSWORD" "$UEFI_HOST_PASSWORD"; do
 done
 info "image: ${MAT_IMAGE_REPO}:${MAT_IMAGE_TAG}   hosts: ${HOST_COUNT}   dpus/host: ${DPU_PER_HOST}"
 
+# GOTCHA: Postgres out-of-memory does not degrade gracefully. The kernel kills a
+# backend, the postmaster crash-recovers, and every connection is dropped with
+# in-flight transactions rolled back. Downstream that presents as stalled machine
+# readiness, controller panics and deadlocks, so the real cause is easy to miss.
+# A 4,500-host run against the old 4Gi default logged 531 OOM kills and never
+# completed; the same run at 24Gi peaked at ~8.7GiB with zero kills. Warn here so
+# an undersized database is visible at minute zero rather than hour three.
+_pg_mem="$(kubectl get postgresql -n "${POSTGRES_NS:-postgres}" nico-pg-cluster \
+    -o jsonpath='{.spec.resources.limits.memory}' 2>/dev/null || true)"
+if [[ -n "$_pg_mem" ]]; then
+    _pg_gib="${_pg_mem%Gi}"
+    if [[ "$_pg_mem" == *Mi ]]; then _pg_gib=$(( ${_pg_mem%Mi} / 1024 )); fi
+    if [[ "$_pg_gib" =~ ^[0-9]+$ ]]; then
+        # Same rough scale as the guidance in helm-prereqs/values.yaml.
+        _pg_want=4
+        (( HOST_COUNT > 500 ))   && _pg_want=8
+        (( HOST_COUNT > 1500 ))  && _pg_want=16
+        (( HOST_COUNT > 5000 ))  && _pg_want=32
+        if (( _pg_gib < _pg_want )); then
+            warn "postgres memory limit is ${_pg_mem} but ${HOST_COUNT} hosts wants >= ${_pg_want}Gi"
+            warn "  undersized postgres OOM-kills under load and breaks ingestion in ways that look unrelated"
+            warn "  raise postgresql.resources.limits.memory in helm-prereqs/values.yaml and reinstall"
+            warn "  (a resize applied before teardown is reverted: teardown deletes the postgres namespace)"
+        else
+            ok "postgres memory ${_pg_mem} is adequate for ${HOST_COUNT} hosts (>= ${_pg_want}Gi)"
+        fi
+    fi
+fi
+
 # =============================================================================
 # Phase 1 — namespace
 # =============================================================================

helm-prereqs/values.yaml

@@ -193,11 +193,39 @@ postgresql:
   ## Points at the operator-managed Service in the postgres namespace.
   host: "nico-pg-cluster.postgres.svc.cluster.local"
   ## Resource limits for each postgres instance.
-  ## Production uses 32 CPU / 16Gi — tune down for dev clusters.
+  ##
+  ## SIZE THIS TO THE FLEET. Postgres running out of memory does not degrade
+  ## gracefully: the kernel OOM-kills a backend, the postmaster performs a full
+  ## crash-recovery, and every open connection is dropped with in-flight
+  ## transactions rolled back. Downstream that looks like unrelated defects --
+  ## stalled machine readiness, controller panics, deadlocks -- so the real
+  ## cause is easy to miss.
+  ##
+  ## Measured on a 4,500-host (13,500 machine) ingestion:
+  ##   4Gi limit  -> 99.96% memory, 531 OOM kills, 112 crash-recoveries,
+  ##                 ingestion unable to complete
+  ##   24Gi limit -> ~8.7GiB peak, zero OOM kills, full fleet ingested and
+  ##                 every machine reaching ready
+  ##
+  ## Rough guidance (limits.memory), assuming ~3 machines per host:
+  ##   up to   500 hosts   4Gi
+  ##   up to 1,500 hosts   8Gi
+  ##   up to 5,000 hosts  16Gi   <- default below
+  ##   beyond              32Gi, and measure
+  ##
+  ## To check a running cluster:
+  ##   kubectl exec -n postgres <pod> -c postgres -- sh -c \
+  ##     'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max; \
+  ##      grep oom_kill /sys/fs/cgroup/memory.events'
+  ## A non-zero and rising oom_kill count means everything downstream is a
+  ## symptom of this.
+  ##
+  ## NOTE: teardown deletes the postgres namespace, so a resize applied before
+  ## a reinstall is silently reverted. Change it here, not on the live cluster.
   resources:
     limits:
-      cpu: "4"
-      memory: "4Gi"
+      cpu: "8"
+      memory: "16Gi"
     requests:
-      cpu: "500m"
-      memory: "1Gi"
+      cpu: "2"
+      memory: "4Gi"