fix(runner/triggers): treat Forbidden as 'no rollouts' in service_no_endpoints workload sweep - #539
Merged
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the service backends lister to treat both NotFound and Forbidden errors as "no rollouts" when querying Argo Rollouts, preventing workload probe failures on clusters without Argo or proper RBAC. The reviewer suggests caching this "unsupported" status after the first failure to avoid repeated API calls that could overload the API server and flood audit logs with 403/404 errors.
Contributor
|
📦 Image Tags Updated |
1 similar comment
Contributor
|
📦 Image Tags Updated |
hsundar-git
previously approved these changes
Jul 15, 2026
Contributor
|
📦 Image Tags Updated |
Clusters without the Argo CRD return 403 (RBAC denies before resource existence is checked; the chart gates rollouts RBAC on the CRD existing at install time), not 404. Propagating it as an error made the predicate fail open and suppressed every service_no_endpoints finding on CRD-less clusters — observed live on the dev cluster.
…04 probes Once the rollouts API returns NotFound/Forbidden, remember it so CRD-less clusters don't take a guaranteed failing round-trip (and audit-log entry) on every service check. CRD installed later needs an agent restart — same trade-off the chart makes gating rollouts RBAC on the CRD existing at install time.
…s evidence The investigate page renders per-block insights; the collector's only sources are Warning rows in event tables (a zero-endpoint Service emits no K8s events) and pod-status extraction from the raw-event json (the subject is a Service), so these findings surfaced with an empty Insights section. Ship the insight explicitly via additional_info .insights, which the collector passes through verbatim.
mayankpande88
force-pushed
the
fix/service-no-endpoints-forbidden
branch
from
July 16, 2026 10:58
7c9e971 to
3050f2e
Compare
Contributor
|
📦 Image Tags Updated |
Contributor
|
📦 Image Tags Updated |
blue4209211
approved these changes
Jul 20, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On clusters without the Argo Rollouts CRD,
service_no_endpointsnever fires. Found during live verification on the dev cluster: the selector-mismatch repro produced zero findings despite kubewatch delivering the Service update.Root cause: the chart only grants rollouts RBAC when the CRD exists at install time, and the apiserver's authorizer rejects the request before checking resource existence — so CRD-less clusters return 403 Forbidden, not 404 NotFound. The rollouts sweep propagated the 403 as an error, and the predicate's fail-open (never alert on missing data) suppressed every finding.
Fix
Treat
ForbiddenlikeNotFoundin the rollouts sweep: best-effort, counts as "no rollouts". The typed sweeps (Deployments/StatefulSets/DaemonSets) already succeeded at that point; only genuinely unexpected errors still fail open.Verification
6d1cadc): Service selector patched to a dead label → endpoints gone → no finding,alerts_forwarded_total 0, runner boot log shows the same 403 on its rollout discovery probe.