[FEATURE] [LFX] Headlamp plugin for node-readiness-controller

[ajaysundark]

Is your feature request related to a problem or existing issue? Please describe.

NRC state today is only visible via kubectl, logs, and the Prometheus metrics surface. It may be desirable for the Headlamp users to get answers to "why is this node not accepting workloads?" directly from the UX.

This issue tracks building a plugin for Headlamp to visualize NRC state directly, proposed as a CNCF LFX mentorship project (2026 Term 3). This is a follow-up deliverable from the Term 2 observability project.

Describe the solution you'd like

High-level scope:

• Cluster-level view of NodeReadinessRule objects with live status (nodes matched / held / released per rule, derived from node taints).
• Node-level drill-down: applicable rules, per-condition pass/fail with failure reasons, NRC-managed taints, and related Events.
• Watch-based, API-state only: This doesnt replace the metrics / Grafana. Headlamp will give visibility into per-object questions and capture relationships on the map.
• Align with other standardized node conditions to explore as a generic lifecycle-conditions panel for Node Availability.
• Design doc reviewed by NRC and Headlamp maintainers before implementation.
• Validate and document the UX and scale tests / limits for the plugin.
• Published to Headlamp Artifact Hub with docs.

Describe alternatives you've considered

No response

[ajaysundark]

This for LFX Term3 mentorship project, please refrain from self-assigning the feature.

cc @AvineshTripathi @AnuragThePathak @illume

[soumadipcodeEnthusiast]

Thanks for opening this — not self-assigning, just a couple of scoping questions ahead of Term 3 applications:

On generalizing to other node conditions beyond NRC — should the initial build stay NRC-specific (NodeReadinessRule + managed taints), with the generic lifecycle-conditions panel as a stretch goal? Or generic from day one?
Since this follows the Term 2 observability project — is there an existing design doc or plugin skeleton from that term worth reading first, or is this a clean start?

[LarytheLord]

I have been reading toward this and two open PRs change what the plugin should be built on, so sharing before I write anything longer.

#344 says per-node lists are being removed from NodeReadinessRule.status for scalability, and that per-node drill-downs move to the new NodeReadinessEvaluation CRD in #345. That matters here because the natural way to build the node view, listing rules and reading appliedNodes and failedNodes, would be building on something you are deleting. So the data sources look like they split by scope rather than by screen:

• fleet rollups from the metrics in docs: add observability design #344, since those are the only place that stays cheap at scale
• per-rule health from NodeReadinessRule.status
• per-node drill-down from a single NodeReadinessEvaluation get, which replaces a client-side join across every rule
• lifecycle history from Node events, which already exist. TaintAdded, TaintRemoved and TaintAdopted are emitted against the Node today in node_controller.go and nodereadinessrule_controller.go, and docs: add observability design #344 adds BootstrapCompleted, so the drill-down needs no controller changes to show history

The NRE timestamps also answer time in state directly rather than by inference. firstEvaluatedAt, taintObservedAt, taintAddedAt and taintRemovedAt give hold duration per rule per node, which is the thing an operator actually asks when a node is not accepting workloads.

My question is about the feature flag. NRE is opt-in behind --enable-node-readiness-evaluation. Does the plugin require it, or does it degrade when the flag is off? Requiring it keeps the plugin simple and matches where the project is going. Degrading means keeping a fallback path that reconstructs per-node state from rule status, which is the code you are removing, so it would be carrying the old model forward inside the UI. I lean towards requiring NRE and showing an explicit empty state that names the flag, but that is a product call rather than a technical one.

On NRC-specific versus the generic lifecycle-conditions panel, I would build NRC-specific first with the condition reading behind one interface, so generic becomes a swap rather than a rewrite. #344 noting that kube_node_status_condition already exports custom conditions through kube-state-metrics suggests the generic view has a data source that does not depend on NRC at all, which supports keeping the two separate from the start.

[DsThakurRawat]

Not self-assigning — a datasource question that I think decides the plugin's architecture, and I could not find it answered in #344 or the thread.

#344 routes every fleet rollup to Prometheus: node_readiness_rule_nodes{rule,state}, node_readiness_rule_matched_nodes and node_readiness_blocked_nodes are all served by the new scrape-time collector, explicitly because per-node lists are leaving NodeReadinessRule.status. That is clearly right for a Grafana SLO view.

But a Headlamp plugin runs in the browser and reaches the cluster through Headlamp's API proxy. In the Headlamp tree, frontend/src/lib/k8s/api/v1/metricsApi.ts fetches metrics with clusterRequest against /apis/metrics.k8s.io/v1beta1/... — an aggregated API served by the apiserver, not a scrape of a controller's /metrics. The in-tree plugins/examples/resource-charts builds its rollup client-side from K8s.ResourceClasses.Pod.useList(). The only Prometheus integration I can find is a catalog-installed plugin rather than something core, so a plugin cannot assume a Prometheus datasource is present or reachable.

That leaves two branches, and they lead to different plugins:

1. The plugin depends on Prometheus. Fleet numbers match the dashboard exactly, but the plugin inherits a hard dependency plus a proxy and auth path to reach the endpoint, and it shows an empty state on clusters that do not have it.
2. The plugin computes rollups from the API. No new dependency, but the aggregate it would read from NodeReadinessRule.status is what docs: add observability design #344 removes, and rebuilding it from NodeReadinessEvaluation means a LIST across every node — the client-side join ✨ feat: introduce NodeReadinessEvaluation (NRE) CRD and controller #345 is meant to eliminate.

So my question is which one you intend. A third reading is that the split is deliberate: Grafana owns the fleet SLO view under #182, and the plugin owns what the API can answer well — per-rule health, per-node drill-down from a single NRE get, and lifecycle from the Node events that already exist. If that is the intent, it would be worth stating in #327, because both branches above are reasonable to build toward and they produce different plugins.

This also compounds the feature-flag question above: NodeReadinessEvaluation is opt-in behind --enable-node-readiness-evaluation, so a cluster can be missing the per-node source and the fleet source independently. Knowing which of those the plugin is allowed to require would settle the empty states before any code exists.

[LarytheLord]

@DsThakurRawat I went and read the observability design doc itself rather than just the PR
summary, and think it answers part of this. Its Goals are "add granular per-rule metrics"
and "build a Grafana dashboard for fleet-wide SLOs," and its Non-Goals say plainly: "Build a
Headlamp plugin. This was originally a goal, but we rescoped it as a follow-up project for
LFX Term 3." If that reading is right, the plugin would not need Prometheus for the fleet-SLO
view specifically, since that view is explicitly someone else's scope, and per-node would
come from a single NodeReadinessEvaluation get either way.

Where I am less sure is per-rule counts without Prometheus. NodeReadinessRuleStatus on
main still carries appliedNodes and failedNodes today, but I would call both shaky
rather than free right now: #341 (open, reproduced) shows updateRuleStatus can silently
lose concurrent per-node writes to either field with no error or retry, and #381 (open) shows
failedNodes separately getting stuck with stale entries that never clear on their own. The
design doc also proposes removing both once the scrape-time collector ships, so today's
counts are not just scheduled to go away, they are not fully trustworthy yet either. One
small thing worth knowing while reading #345: its NodeReadinessEvaluationReconciler is
deliberately only eventually consistent with the Node it derives from (per Karthik-K-N's
reply this morning), so a per-node NRE read can briefly lag the Node object it is summarizing.
Not weighing in on that design, just flagging it in case the plugin side had not seen it yet.

@ajaysundark, the question underneath all of this is one only you can really settle. v0.5.0
is cut August 14th per #377, and its changelog includes neither #344 nor #345, and #345
itself is still being discussed as of this morning. So whoever starts this in September is
choosing between two different twelve weeks: build the plugin's first weeks around helping
land NRE and the metrics work, or scope entirely against what exists on main today
(appliedNodes/failedNodes, no per-node CRD) and treat NRE as an enhancement to bolt on
once it ships. Which of those is closer to how you would want this scoped, or is that itself
still undecided on your side?

[DsThakurRawat]

@LarytheLord one correction on the second of the two counts you called shaky: #381 is the fix, not the symptom. the stale failedNodes bug it addresses is #376, and i ran the pr branch today (873d145, which includes the assertion fix from this afternoon) against that repro: the stale entry clears from the persisted rule status and the full suite is green. so that half is a fix pending review rather than an open problem. can't speak to #341, i haven't tried its repro.

[tejassinghbhati]

I guess that there is one thing on using Node events for the lifecycle history.
Worth knowing that TaintAdded / TaintRemoved / TaintAdopted were not actually being recorded on Helm installs until #351 landed last week. The chart's ClusterRole was missing the events.k8s.io group, so every one of those writes came back forbidden and nothing showed up. On any cluster installed from the chart before that, the history the plugin would read is just empty.

The bigger thing is retention. Events default to around an hour TTL, so they answer "what happened recently" but not "how long has this node been held". The NRE timestamps are the better source for that, and @ajaysundark 's own point on #384 was that events are best-effort by design, so I would not lean on them as the history store.