Skip to content

fix(scanners): own copied pull-secrets at create time so they GC with the Job - #505

Merged
blue4209211 merged 2 commits into
mainfrom
fix/scanner-pull-secret-ownerref
Jul 4, 2026
Merged

fix(scanners): own copied pull-secrets at create time so they GC with the Job#505
blue4209211 merged 2 commits into
mainfrom
fix/scanner-pull-secret-ownerref

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Description

Copied image-pull secrets (imgps-*) were leaking: they accumulate in the scanner namespace and never get cleaned up.

Root cause

The copy flow created each Secret without an ownerReference, then tried to attach one via a post-create Secret Update (ownReferencedSecrets). But the runner ServiceAccount has create on secrets and not update/delete. Additionally, the ownerReference set BlockOwnerDeletion: true, which the GC admission plugin only allows if the caller has update on the owner's finalizers subresource (jobs/finalizers) — which the agent also lacks. So the Update was rejected (403) and, being best-effort, silently dropped. Result: every copied secret was orphaned with no ownerReference, so neither the Job's TTL nor the reaper ever garbage-collected it.

(Confirmed on a live cluster: kubectl auth can-i update secrets → no, update jobs/finalizers → no; observed imgps-* secrets with empty ownerReferences.)

Fix

Stamp the ownerReference at Secret create time (needs only create, which the agent has). Since the reference needs the Job UID and the pod must not pull before its credentials exist:

  1. Resolve the pull secrets (read source, build the copy objects — don't create).
  2. Create the Job suspended, referencing the not-yet-created copies on its pod.
  3. Create the copies owned by the Job.
  4. Resume the Job (patch spec.suspend=falsepatch on jobs is already granted).

BlockOwnerDeletion is dropped (unnecessary for GC, and avoids the jobs/finalizers requirement). If resume fails, the Job is deleted so its owned copies GC with it instead of lingering.

No RBAC/chart change, and no expansion of the agent's secret permissions (a customer-deployed agent stays at create-only on secrets).

Testing

  • Unit test (TestAutoCopyPullSecrets_CopiesRegistrySecretsAndAttaches): asserts each copy is Job-owned and the Job is not left suspended.
  • Integration test against a real apiserver (TestPrimitives_RealAPIServer/ScheduleJob_AutoCopyPullSecrets): confirms the copy is owned by the Job end to end. Both pass; go build ./..., go vet, gofmt clean.

Notes

  • Non-pull-secret scans are unchanged: the Job is only suspended when there are pending copies to create.
  • Pairs with the api-server manual-scan tenant fix and the SCANNER_AUTO_COPY_PULL_SECRETS enablement (separate PRs) that make private-image scans work in the first place.

… the Job

image-scan pull-secret copies were created without an ownerReference and then
patched with one via a post-create Secret Update. The runner ServiceAccount has
`create` but not `update`/`delete` on secrets (and setting BlockOwnerDeletion
additionally needs `update` on jobs/finalizers, which it also lacks), so the
Update was rejected and silently dropped — every copied `imgps-*` secret was
left orphaned in the scanner namespace, accumulating forever.

Stamp the ownerReference at Secret *create* time instead (needs only `create`,
which the agent has). Because the reference needs the Job UID, and the pod must
not pull before its credentials exist, create the Job suspended, create the
copies owned by it, then resume it (patch spec.suspend=false — `patch` on jobs
is already granted). BlockOwnerDeletion is dropped (unnecessary for GC and would
require jobs/finalizers). If resume fails, the Job is deleted so its owned copies
GC with it rather than lingering.

No RBAC/chart change and no expansion of the agent's secret permissions.

Verified: unit test asserts the copy is Job-owned and the Job is not left
suspended; the real-apiserver integration test (ScheduleJob_AutoCopyPullSecrets)
confirms ownership end to end.
@mayankpande88
mayankpande88 requested a review from a team as a code owner July 4, 2026 07:52

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Kubernetes Job creation flow to safely handle image-pull secrets. Instead of copying secrets beforehand, the Job is created in a suspended state, the copied secrets are created with the Job's UID as the owner reference, and then the Job is resumed. A review comment points out a critical issue in the error handling path: if resuming the Job fails due to context cancellation or timeout, the cleanup Delete call will also fail because it uses the same cancelled context. It is recommended to use a non-cancelled context (such as context.WithoutCancel(ctx)) for the cleanup deletion and log any failures.

Comment thread runner/pkg/scanners/primitives.go
resumeJob most often fails because ctx was cancelled/timed out; the cleanup
Delete reused that ctx and would fail immediately, leaking the suspended Job.
Detach via context.WithoutCancel (bounded to 15s) and log a delete failure.

Addresses PR review feedback.
@blue4209211
blue4209211 merged commit 16e73b6 into main Jul 4, 2026
7 checks passed
@blue4209211
blue4209211 deleted the fix/scanner-pull-secret-ownerref branch July 4, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants