fix(agent): detect single-OOM pods and add node/pod condition matchers - #500
Merged
Merged
Conversation
OOM detection only fired on lastState.terminated, which a container populates after a restart. A restartPolicy:Never pod or a Job that OOMs once records the kill in state.terminated with an empty lastState, so it was never detected. The pod_oom_killed predicate now fires off either state or lastState (via mostRecentOOMKilledContainerStatus, already used by the enricher); the dead lastState-only helper is removed. Adds three Node/Pod matchers that read the watched object directly, so they survive a degraded Prometheus rule engine: - node_unschedulable: a node cordoned for >15m (no Prometheus equivalent) - node_pressure: Disk/Memory/PID pressure on a node - pod_unschedulable: PodScheduled=False (no node fits) for >10m convertNode now sends spec.unschedulable so the cordon state is available downstream. extra-scrape-config.yaml drops the additionalPrometheusRulesMap block, which fully duplicated the agent chart's prometheus-alert-rule.yaml.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request removes several Prometheus rules from the scrape configuration and replaces them with built-in agent matchers for unschedulable nodes, node pressure, and unschedulable pods. It also refactors the OOMKilled matcher to support pods with a 'Never' restart policy. The review feedback identifies a critical issue in the node unschedulable matcher where a missing or unparseable taint timestamp causes immediate alerts, bypassing the intended 15-minute grace period. A code suggestion is provided to return false in these scenarios, and the corresponding test should be updated.
Contributor
|
📦 Image Tags Updated |
…estamp is present The node.kubernetes.io/unschedulable taint is added asynchronously after spec.unschedulable flips, so it's absent on the first cordon update. Treating a missing timeAdded as sustained fired the matcher immediately, bypassing the 15m grace and alerting on routine drains/upgrades. Return false until the taint timestamp can confirm the cordon outlasted the window.
Contributor
|
📦 Image Tags Updated |
blue4209211
approved these changes
Jun 27, 2026
2 tasks
blue4209211
pushed a commit
that referenced
this pull request
Jun 29, 2026
* chore(chart): bump nudgebee-agent chart version to 0.1.5 nudgebee-agent-0.1.4 is already released/tagged (2026-06-25, before the node/pod matcher work landed), so the full-release chart-releaser run 422'd on the existing tag. Bump the chart version so the release cuts a fresh nudgebee-agent-0.1.5 carrying #500 (matchers) + #501 (alert subject resolution). appVersion left as-is (decoupled; release pins runner image by GHCR tag, not appVersion). * chore: update image tags for main release --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two detection gaps in the agent.
OOM on
restartPolicy:Never/ Jobs. Thepod_oom_killedpredicate only checkedlastState.terminated, which a container populates after a restart. A pod that OOMs once and never restarts records the kill instate.terminatedwith an emptylastState, so it was never detected. The predicate now fires offstateorlastStateviamostRecentOOMKilledContainerStatus(already used by the enricher); the dead lastState-only helper is removed.Three new Node/Pod matchers that read the watched object directly (independent of Prometheus rule evaluation):
node_unschedulable— a node cordoned (spec.unschedulable) for >15m. No Prometheus equivalent.node_pressure— Disk/Memory/PID pressure on a node.pod_unschedulable—PodScheduled=False(no node fits) sustained >10m. Slow-start pods (PodScheduled=True) are excluded.Discovery:
convertNodenow sendsspec.unschedulableso cordon state is available downstream.extra-scrape-config.yaml: drops theadditionalPrometheusRulesMapblock, which fully duplicated the agent chart'sprometheus-alert-rule.yaml(the chart versions use better ratio/baseline expressions);KubeHpaMaxedOutwas being defined twice.Tests
predicates_test.go: OOMstate.terminatedregression + 10 tests for the new matchersconverters_test.go:unschedulableassertiongo test ./pkg/triggers/ ./pkg/discovery/green; fmt/vet cleanNote for reviewers
node_pressureoverlaps the upstreamKubeNodePressurerule andpod_unschedulablepartially overlapsKubePodNotReady. Until those upstream rules are disabled they will double-fire; that retirement is left as a follow-up decision.