fix: stop forwarding WatchList options to internal CRD watch#1845
Conversation
The packaging aggregated API server is a thin proxy whose storage is the host cluster's InternalPackage(Metadata) CRDs, and its custom REST watch does not synthesize the WatchList "initial-events-end" bookmark. With WatchList enabled, the embedded apiserver's SetListOptionsDefaults injects sendInitialEvents + resourceVersionMatch=NotOlderThan into watch requests with resourceVersion=0. Host clusters that ship the WatchList feature gate disabled (e.g. Kubernetes 1.33, which turned it off by default) then reject the forwarded options with 422 Unprocessable Entity, failing a plain "watch&resourceVersion=0" (TestK8APIEndpoints). Disabling the feature gate in the packaging apiserver process fixes this across every host version: raw watch&resourceVersion=0 requests are served as a plain watch (no injection), and WatchList-capable clients (client-go informers) receive a 422 for the streaming request and transparently fall back to list+watch. Note: an earlier approach that stripped these options only in the REST watch path was rejected because it returned a successful watch stream with no bookmark, which hangs WatchList informers (including kapp-controller's own) and stalls the shared manager cache. Signed-off-by: Sameer <sameer.khan@broadcom.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Sameer <sameer.khan@broadcom.com>
f606bfe to
e05009b
Compare
himsngh
left a comment
himsngh
left a comment
There was a problem hiding this comment.
Approving this. I traced the vendored code to validate the mechanics: SetListOptionsDefaults returns early when the gate is off, which guarantees sendInitialEvents isn't injected. The upstream gate history (on in 1.32, off in 1.33, on in 1.34) also cleanly explains the version-specific 422s we are seeing.
I also validated the blast radius and fallback behavior. Flipping this globally won't affect our own outbound host cluster watches since client-go uses a separate feature flag, and relying on a 422 to trigger a list+watch is the explicitly designed fallback path for the upstream reflector.
Let's make sure we open an issue to expand our CI version matrix so we can catch these types of cross-version API failures moving forward.
There was a problem hiding this comment.
Pull request overview
Disables the Kubernetes WatchList feature gate inside kapp-controller’s aggregated packaging API server to avoid forwarding sendInitialEvents defaults into the host cluster’s CRD watch requests, which are rejected (422) on Kubernetes 1.33 where WatchList is disabled by default.
Changes:
- Force-disable the
WatchListfeature gate at packaging apiserver startup to prevent defaultingsendInitialEventsonwatch=1&resourceVersion=0. - Add a unit test asserting the packaging apiserver runs with
WatchListdisabled.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/apiserver/apiserver.go | Disables WatchList feature gate during packaging apiserver initialization. |
| pkg/apiserver/apiserver_test.go | Adds a test validating WatchList is disabled for the packaging apiserver. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func Test_disableWatchListFeatureGate(t *testing.T) { | ||
| err := disableWatchListFeatureGate() | ||
| require.NoError(t, err) | ||
|
|
||
| require.False(t, utilfeature.DefaultFeatureGate.Enabled(genericfeatures.WatchList), | ||
| "WatchList feature gate must be disabled for the packaging apiserver") | ||
| } |
What this PR does / why we need it:
The packaging API server enables the
WatchListfeature by default, so it injectssendInitialEventsintowatch=1&resourceVersion=0requests and forwards them to the host cluster's watch on the backing CRDs. On Kubernetes 1.33 theWatchListgate is disabled by default, so the host rejects those requests with 422 Unprocessable Entity (gate default flipped off in 1.33, back on in 1.34).This disables the WatchList feature gate in the packaging API server. Raw
watch=1&resourceVersion=0requests are then served as a plain watch (no injection), and WatchList-capable clients cleanly fall back to list+watch — consistent across all host versions.Manual Validation
This fix was validated by reproducing the issue and confirming the fix on two isolated
kindclusters.Environment
Both clusters run Kubernetes v1.33.4 — the only affected version, since the
WatchListfeature gate ships disabled by default on 1.33 (kube_features.go#L415-L420). Confirmed on both clusters:Two kapp-controller images were built from the same Dockerfile — one from
develop(pre-fix), one from this branch (post-fix) — and deployed one per cluster. Both pods reported2/2 Runningwith the packagingAPIServiceAvailable=True, so the kapp-controller build was the only variable between the two clusters.Request
Results
kc-nofixdevelop@721929598(no fix)422 Unprocessable Entitykc-fixe05009b71(with fix)200 OKBefore fix:
After fix:
Validation & Compatibility Matrix
To ensure absolute backward compatibility and stability across our support window following the
v0.36.0library upgrades, the complete test suite (Unit, Validation, and E2E integration tests) was successfully executed against the following Kubernetes environments:test-gh / Run all testsv1.32.13test-gh / Run all testsv1.33.12test-gh / Run all testsv1.34.8test-gh / Run all testsv1.35.5test-gh / Run all testsv1.36.1Kind Cluster E2E testskinder)v1.36.x