Slice 6: Assert webhook receiver reachability in-cluster via the Rancher proxy (monitoring airgap)

[floatingman]

Part of #848 (PRD: Make Monitoring and Alerting Validation Tests Airgap-Compatible). Follow-up slice closing a PRD gap discovered during Slice 1 (#849) live verification.

Problem

The webhook receiver accessibility check in validation/charts/monitoring_test.go (TestMonitoringChart) ends with a runner-side GET: ingresses.IsIngressExternallyAccessible(client, "<nodeAddr>:<nodePort>", "dashboard", false) dials the selected node address directly from wherever go test executes.

The PRD's node-address work (Slice 1) solves address selection — falling back to InternalIP when a private VPC offers no external addresses — but nothing in #848 addresses runner-side routing. Two live verification runs against a real airgap cluster (#858) confirmed: every registry/image/address gate passes, and the run fails deterministically at exactly this line (context deadline exceeded to 172.31.255.x) because the runner has no route into the node subnet.

This is structural for CI, not a one-off:

• Jenkinsfile.airgap-rke2-tests executes go test in a Docker container on the Jenkins agent, reaching Rancher through the public ELB hostname. The agent is never inside the airgap network; the bastion is the only public entry point.
• Airgap clusters expose only internal node addresses, so no nodeAddressPreference can produce a runner-reachable address.
• NeuVector airgap CI passes from the same runner position because every check goes through the Rancher API proxy; this check is the only remaining direct runner→node HTTP call in the suite.

Consequence: the combined airgap job (Slice 4, #852) would go red on this line forever, forcing permanent use of skipWebhookReceiver — which zeroes the alerting end-to-end coverage that is the PRD's headline feature.

Solution

Move the assertion in-cluster: execute the reachability probe via the Rancher-managed plane instead of the runner's network position, using the same mechanism teardown already trusts — kubectl.Command (shepherd extensions/kubectl), which provisions a short-lived job running the shell-image container (has curl) on the target cluster through the Rancher proxy. Zero pipeline changes; lives entirely in this repo.

Semantics: "externally accessible from the runner" becomes "cluster-internal reachable" — for an airgap target arguably the correct assertion, since there is no "external" in airgap. Bonus: removes runner egress entirely, aligning with the PRD's egress-audit goal (Story 20).

Non-airgap behavior is preserved by keeping both probes where they make sense:

• When the resolver selected an ExternalIP (non-airgap default), the existing runner-side check still runs — public addresses are runner-reachable there, so outcomes are unchanged (PRD Story 11).
• When the resolver fell back to InternalIP, the runner-side check is skipped (it can only test routing luck, not the receiver) and the in-cluster probe is authoritative.

Tasks

• Add an in-cluster HTTP probe helper (e.g. actions/monitoring or the charts helpers) wrapping kubectl.Command: curl -sS -o /dev/null -w '%{http_code}' http://<nodeAddr>:<nodePort>/dashboard against the target cluster, asserting an expected status
• Replace the unconditional runner-side IsIngressExternallyAccessible assert in TestMonitoringChart with the address-type-conditional logic above
• Unit-test the pure pieces (URL construction, address-type → probe-mode decision, status-code classification); the kubectl.Command call is exercised by the live run
• Update validation/charts/README.md: the "runner must be inside the network" note is replaced by the in-cluster probe description; document any new config (none expected)
• Re-verify live from a runner outside the airgap network

Acceptance Criteria

• Webhook receiver accessibility is asserted from inside the target cluster via the Rancher proxy (kubectl.Command / shell-image); no direct runner→node HTTP on the internal-address path
• Non-airgap runs with an ExternalIP selection keep the existing runner-side assertion — same pass/fail outcomes as before
• A live airgap run executed from a runner outside the airgap network passes the full webhook receiver end-to-end section without setting skipWebhookReceiver
• Unit tests pass for the new pure logic; go vet / build with validation tags clean
• validation/charts/README.md updated accordingly

Out of Scope

• Pipeline/Jenkinsfile changes ( bastion HTTP proxy for the runner is the alternative design, rejected here to keep the pipeline reused unchanged per PRD)
• Changing the alertmanager→receiver delivery path (already in-cluster and working)
• The DeleteMonitoringResources teardown server-url DNS issue observed on the hand-built verification env (separate environment defect, not CI-relevant: CI-provisioned Rancher uses the public ELB name)

Dependencies

• Slice 1 (Slice 1: Configurable pinned webhook receiver image, registry resolution, and node address resolver (monitoring airgap) #849): consumes the node address resolver and its address-type information

Effort

0.5–1 day