Add Scale Test

[vitorfloriano]

Description

This PR adds a scale test harness for Node Readiness Controller. The test harness runs on a single rule on a 1000 nodes by default, and offers different tuning parameters for QPS, Burst, concurrent workers, among others.

Benchmarking plan:

In this first iteration, we test how NRC performs when reconciling a single NodeReadinessRule in continuous mode to 1000 nodes (default node count).
We switch conditions to true and false to observe the controller adding and removing taints from nodes.

Next iterations can include more test cases, like:

• Different node counts
• Different rule counts
• Bootstrap-only mode
• Different runtimes (kind, podman)

We also selected the metrics we deemed most relevant for the scalability report.

Here's a list of all metrics exposed/scraped during the test run:

available metrics at the controller endpoint

• Custom Controller Metrics (Node Readiness Specific)

  • node_readiness_condition_failures_total: Total number of failed condition evaluations by rule and condition name.
  • node_readiness_evaluation_duration_seconds_bucket / _count / _sum: Histogram of the duration (in seconds) taken
  to evaluate rules.
  • node_readiness_reconciliation_latency_seconds_bucket / _count / _sum: Histogram of total node readiness
  reconciliation latency.
  • node_readiness_rule_last_reconciliation_timestamp_seconds: Timestamp (Unix epoch) when a rule was last reconciled.
  • node_readiness_rules_total: Total number of active node readiness rules.
  • node_readiness_taint_operations_total: Total number of taint operations (add/remove) executed.

• controller-runtime Core Metrics

  • controller_runtime_active_workers: Number of active reconciler workers.
  • controller_runtime_max_concurrent_reconciles: Configured maximum concurrent reconciles for a controller.
  • controller_runtime_reconcile_total: Total number of reconciliations.
  • controller_runtime_reconcile_errors_total: Total number of reconciliation errors.
  • controller_runtime_reconcile_panics_total: Total number of reconciler panics.
  • controller_runtime_reconcile_time_seconds_bucket / _count / _sum: Histogram of time spent on reconciliations.
  • controller_runtime_conversion_webhook_panics_total / controller_runtime_webhook_panics_total: Total panics inside
  conversion or admission webhooks.
  • controller_runtime_terminal_reconcile_errors_total: Total terminal reconciliation errors.

• client-go Workqueue Metrics

  • workqueue_adds_total: Total number of tasks added to the workqueue.
  • workqueue_depth: Current queue depth.
  • workqueue_longest_running_processor_seconds: Processing time of the longest running task.
  • workqueue_queue_duration_seconds_bucket / _count / _sum: Histogram of how long items sit in the workqueue before
  processing.
  • workqueue_work_duration_seconds_bucket / _count / _sum: Histogram of how long processing an item takes.
  • workqueue_retries_total: Total number of task retries.
  • workqueue_unfinished_work_seconds: Seconds spent on currently running unfinished tasks.

• client-go REST Client Metrics

  • rest_client_requests_total: Total HTTP requests made by the Kubernetes API client.

• OS Process Metrics (Host Resources)

  • process_cpu_seconds_total: Total CPU user and system time spent in seconds.
  • process_resident_memory_bytes: Resident memory size (RSS) in bytes.
  • process_virtual_memory_bytes / _max_bytes: Virtual memory size.
  • process_open_fds / _max_fds: Number of open file descriptors.
  • process_network_receive_bytes_total / _transmit_bytes_total: Network traffic bytes.
  • process_start_time_seconds: Start time of the process.

• Go Runtime Internal Metrics

  • go_goroutines: Number of active goroutines.
  • go_threads: Number of OS threads.
  • go_info: Information about the Go runtime version.
  • go_memstats_alloc_bytes / go_memstats_sys_bytes: Go heap memory stats.
  • go_gc_duration_seconds / _count / _sum: Garbage collector cycles durations.
  • go_sched_latencies_seconds_bucket: Go scheduler queue latency histogram.

• Prometheus Scraper Metrics

  • scrape_duration_seconds: Time taken to run the scraper query.
  • scrape_samples_scraped: Number of metric samples retrieved.
  • up: Health status of the scraped controller instance (1 = healthy, 0 = down).

Key design choices:

• Use kwokctl to create the cluster and kwok binary to simulate k8s core components to keep it lightweight (no docker/ kind overhead).
• Run the NRC as a separate binary instead of deploying to the cluster, since that would not be possible with the fake cluster created by Kwok.
• Simulate changes in conditions using Kwok's Stage CRD. See: https://kwok.sigs.k8s.io/docs/user/stages-configuration/
• Use Kwok's native Prometheus integration to keep the setup simple. See: https://kwok.sigs.k8s.io/docs/examples/prometheus/

Dependencies

The test only depends on kwokctl and kubectl being available at PATH. We ensure kwokctl is available and download the binary to hack/tools/bin right at the start of the test, if missing.

The dependency on Prometheus and core components of the control-plane is completely managed by kwokctl.

Testing

I've run the scale test multiple times, with different configurations.

Release Notes

Since this is mostly infra, a release note may not be necessary.

Note

LLM Usage: I used Gemini 3.5 Flash Medium with Antigravity CLI for planning and writing some parts with tricky logic. The code was reviewed by me and manually test multiple times.

Supersedes #185

[kubernetes-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[netlify]

✅ Deploy Preview for node-readiness-controller canceled.

Name	Link
🔨 Latest commit	8136c5a
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a7e93f373d2b40008a8f1a4

[vitorfloriano]

/cc @Karthik-K-N
/cc @ajaysundark
/cc @Priyankasaggu11929

[vitorfloriano]

After much research, this is the initial work. I got a LGTM on the approach from Kwok's maintainer, so I believe I'm headed in the right direction. PTAL.

[vitorfloriano]

@ajaysundark @Priyankasaggu11929 @Karthik-K-N

Following up to our conversation, I'm going to refine the markdown report, perhaps vet some of the metrics there to make it cleaner and easier to draw insights from.

[vitorfloriano]

@Priyankasaggu11929 I addressed your comments from your first round and refactored a bunch of stuff. I also reshaped the template for the report and refactored the test config to make it more concise.

I'll leave the Prom queries out of it for now to let others review.

PTAL.

[ajaysundark]

not actionable now. but I proposed to deprecate this metric in the new metric surface. The reason is that this just counts "condition-not-satisfied" at every iteration, which really are not 'failures'. At scale, this shouldn't bring out bad impression of state of things

[ajaysundark]

/lgtm
/approve

Thanks @vitorfloriano! We want to use this for release pipeline to be able to continuously validate for scale for all the release versions. We should plan to see how we can integrate this into CI with kwok "in-cluster" mode. We can follow up that work in #409

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ajaysundark, vitorfloriano

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ajaysundark]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ajaysundark]

cc @rawadhossain