feat: Add bootstrap NRC duration metric by rawadhossain · Pull Request #294 · kubernetes-sigs/node-readiness-controller

rawadhossain · 2026-07-08T14:33:01Z

Description

This PR adds node_readiness_bootstrap_hold_duration_seconds, which measures how long a node is held by a bootstrap-only rule's taint, from when the taint is first present on the node until bootstrap completes.

The status object now tracks two anchor timestamps on NodeEvaluation:

taintAppliedAt — when NRC itself applies the taint
taintObservedAt — when NRC first observes the taint present, whether applied by NRC or already present (e.g. via kubelet -register-with-taints)

The metric carries a new controller label (applied/adopted) so the two cases are distinguishable: anchored on taintAppliedAt when NRC applied the taint, falling back to taintObservedAt for the adopted case, where taintAppliedAt is never set.

Related to Issue #182

Type of Change

/kind feature

Testing

Unit tests for isBootstrapCompleted, including backward compatibility and in progress states
Integration tests added

Checklist

make test passes
make lint passes

netlify · 2026-07-08T14:33:07Z

✅ Deploy Preview for node-readiness-controller ready!

Name	Link
🔨 Latest commit	`f889aee`
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a86176000cd6f00083cf239
😎 Deploy Preview	https://deploy-preview-294--node-readiness-controller.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

kubernetes-prow · 2026-07-08T14:33:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please assign tallclair for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubernetes-prow · 2026-07-08T14:33:13Z

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ajaysundark · 2026-07-08T17:42:08Z

 )

+// bootstrapAnnotationPayload is the JSON value stored in a bootstrap-completion annotation.
+type bootstrapAnnotationPayload struct {


IMO, TaintAppliedAt and Completed are better suited for the dedicated NodeEvaluation 'status' api we planned. Can we discuss this during our next sync on how we could shape it in the alpha2 api?

AvineshTripathi · 2026-07-20T05:42:00Z

@rawadhossain can you resolve the conflicts please!

rawadhossain · 2026-07-21T19:18:39Z

Hi @ajaysundark @AvineshTripathi
Sorry for the late response.

Updated the PR. Moved the timestamp from node metadata to status.nodeEvaluations, following the current API shape as we discussed. Also handled the adopted taint (--register-with-taints) case by only stamping the anchor when the taint is applied by NRC.

rawadhossain · 2026-07-30T17:52:51Z

While validating this metric for Section 2 Grafana dashboard, I found that it wasn't emitting any data. It turned out TaintAppliedAt in status.nodeEvaluations exposed an issue with stale cached rules in NodeReconciler, which could cause the anchor to be lost during the next status update.

I added a fix to recover the missing anchor when needed and make sure it isn't lost again during later status updates. Since the Section 2 dashboard depends on this metric, I wanted to fix this before moving forward.

cc @ajaysundark @AvineshTripathi

AvineshTripathi · 2026-08-01T05:53:52Z

@rawadhossain can we two separate PRs? Maybe the fix can go in the different PR and we can try merge this. Please let me know your thoughts

ajaysundark · 2026-07-30T21:13:11Z

+	// taintAppliedAt is the timestamp when the controller applied the readiness taint to this Node.
+	//
+	// +optional
+	TaintAppliedAt metav1.Time `json:"taintAppliedAt,omitempty,omitzero"`


cc @Karthik-K-N for note as we should keep this field name and semantics same when we split NodeEvaluation status

rawadhossain · 2026-08-03T19:16:35Z

@rawadhossain can we two separate PRs? Maybe the fix can go in the different PR and we can try merge this. Please let me know your thoughts

That makes sense and its cleaner. My only concern is that the cache fix is tied to this metric, and without it the metric doesn't reliably emit data. That's why I kept them together. If you prefer splitting them, I'm happy to do that.

AvineshTripathi · 2026-08-04T18:50:38Z

@rawadhossain i see there are conflicts, can you resolve it please

rawadhossain · 2026-08-04T21:39:02Z

Done, resolved the merge conflicts. Thanks.

ajaysundark · 2026-08-14T16:31:24Z

+			Help:    "Time from readiness taint application or observation to bootstrap completion. Measures only NRC-attributable hold time.",
+			Buckets: []float64{1, 5, 10, 30, 60, 120, 300, 600, 1200, 1800, 3600},
+		},
+		[]string{"rule", "controller"}, // controller: applied, adopted


can you x-ref the design on this one? taint_origin for this label is better with values adopted vs controller, to show initial ownership.

Updated it. taint_origin now uses controller and adopted to show the taint ownership.

ajaysundark · 2026-08-17T06:03:05Z

please resolve conflicts

rawadhossain · 2026-08-17T18:35:31Z

@ajaysundark resolved the conflicts. PTAL. Thanks.

kubernetes-prow Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 8, 2026

kubernetes-prow Bot requested review from dchen1107 and mrunalp July 8, 2026 14:33

kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 8, 2026

kubernetes-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 8, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from 10d451f to f85970c Compare July 8, 2026 14:51

ajaysundark reviewed Jul 8, 2026

View reviewed changes

kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 10, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from f85970c to a9d441f Compare July 21, 2026 17:25

kubernetes-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 21, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from 3069c56 to 59e4f78 Compare July 21, 2026 17:43

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2026

kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 30, 2026

ajaysundark reviewed Aug 1, 2026

View reviewed changes

kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 1, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from 6f60849 to d72fbc1 Compare August 4, 2026 21:26

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from d72fbc1 to 9832aaa Compare August 4, 2026 21:36

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from 9832aaa to 7456ac8 Compare August 4, 2026 21:43

rawadhossain mentioned this pull request Aug 6, 2026

docs: add observability design #344

Open

2 tasks

ajaysundark added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Aug 7, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from fbed91b to 3966293 Compare August 8, 2026 14:51

tejassinghbhati mentioned this pull request Aug 10, 2026

[BUG] TaintAdopted events are emitted repeatedly for taints the controller applied itself #383

Closed

ajaysundark reviewed Aug 14, 2026

View reviewed changes

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch 2 times, most recently from 8ede5ad to 987eb12 Compare August 15, 2026 14:55

kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 17, 2026

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from 987eb12 to 5163b26 Compare August 17, 2026 13:29

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 17, 2026

extend histogram buckets

f889aee

rawadhossain force-pushed the feat/bootstrap-nrc-duration-metric branch from 5163b26 to f889aee Compare August 19, 2026 20:51

Uh oh!

Conversation

rawadhossain commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related to Issue #182

Type of Change

Testing

Checklist

Uh oh!

netlify Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for node-readiness-controller ready!

Uh oh!

kubernetes-prow Bot commented Jul 8, 2026

Uh oh!

kubernetes-prow Bot commented Jul 8, 2026

Uh oh!

Uh oh!

ajaysundark Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

AvineshTripathi commented Jul 20, 2026

Uh oh!

rawadhossain commented Jul 21, 2026

Uh oh!

rawadhossain commented Jul 30, 2026

Uh oh!

AvineshTripathi commented Aug 1, 2026

Uh oh!

Uh oh!

ajaysundark Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rawadhossain commented Aug 3, 2026

Uh oh!

AvineshTripathi commented Aug 4, 2026

Uh oh!

rawadhossain commented Aug 4, 2026

Uh oh!

ajaysundark Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

rawadhossain Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

ajaysundark commented Aug 17, 2026

Uh oh!

rawadhossain commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rawadhossain commented Jul 8, 2026 •

edited

Loading

netlify Bot commented Jul 8, 2026 •

edited

Loading