fix(kom): recover deletions missed during downtime

[Pybsama]

Summary

• warm each Kubernetes object controller source before the final persisted-state scan
• gate reconciliation workers until every controller has finished state recovery
• use the direct API reader for startup reads while keeping the cached client for normal reconciliation and writes
• publish the normal healthy transition for resources deleted during downtime, while preserving recoverable state on read or publish failure
• decode persisted keys from the right using the REST-mapped GVK scope, so slash-containing policy names remain compatible and removed policies cannot be claimed by a shorter prefix

This preserves the existing exported LoadState(ctx) API; the manager startup
path uses the new scope-aware recovery method.

Closes #1542.

Why the startup ordering matters

A scan before the controller source starts still leaves a gap: an object can be
present during the scan and disappear before the informer's initial list,
producing neither NotFound nor a delete event. The state-loader runnable
therefore waits for the controller's idempotent source warmup first, performs
the direct-API verification second, and only then releases the shared worker
barrier.

If source warmup, state loading, an API read, or recovery publication fails,
Manager.Start returns the error so the workload can retry instead of running
with silently stale quarantine state.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📚 Documentation
• 🔧 Refactoring
• 🔨 Build/CI

Component(s) Affected

• Core Services
• Documentation/CI
• Fault Management
• Health Monitors
• Janitor
• Other: ____________

Testing

• Tests pass locally
• Manual testing completed
• No breaking changes (or documented)

Validation completed on Darwin arm64 with Kubernetes envtest 1.36.2:

• full module race suite: 50 tests passed
• focused startup, barrier, deletion-recovery, publisher-failure, and state-key scope regressions
• go vet ./...
• go build ./...
• repository-pinned golangci-lint v2.5.0: 0 issues
• component make lint-test
• gofmt and git diff --check

The regression coverage includes a real Manager lifecycle, a Pod deleted with
zero grace while the monitor is stopped, direct reads before the manager cache
starts, non-NotFound API errors, recovery publication failure, Node deletion,
cluster- and namespace-scoped slash/prefix collisions, and state left by a
removed policy.

Checklist

• Self-review completed
• Documentation updated (if needed)
• Ready for review

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: eb85a4ce-754d-4612-8aa9-3102061a89d8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

@Pybsama this PR has been inactive for 14 days. Do you need help finishing it, or should we close it for now? Feel free to reopen anytime.