Add Scale Test

.gitignore

@@ -38,3 +38,6 @@ hack/test-workloads/ignore-pending/*
 CLAUDE.md
 GEMINI.md
 
+# Scale test artifacts
+test/scale/artifacts/
+

Makefile

@@ -432,6 +432,16 @@ setup-envtest: $(SETUP_ENVTEST) ## Download the binaries required for ENVTEST in
 	@echo "Setting up envtest binaries for Kubernetes version $(KUBEBUILDER_ENVTEST_KUBERNETES_VERSION)..."
 	@echo KUBEBUILDER_ASSETS=$(KUBEBUILDER_ASSETS)
 
+## --------------------------------------
+## Scale Testing
+## --------------------------------------
+
+##@ scale:
+
+.PHONY: test-scale
+test-scale: manifests generate ## Run the scale performance tests. See test/scale/README.md for more information.
+	go test -tags=scale -v ./test/scale/... -ginkgo.v -count=1
+
 ## --------------------------------------
 ## Hack / Tools
 ## --------------------------------------

test/scale/README.md

@@ -0,0 +1,82 @@
+# Node Readiness Controller Scale Performance Testing
+
+This directory contains the scale performance testing suite for the `node-readiness-controller`. It uses the Kubernetes [KWOK (Kubernetes WithOut Kubelet)](https://kwok.sigs.k8s.io/) cluster simulator to spin up thousands of mock nodes and measure controller performance, throughput, memory footprints, and workqueue latency under heavy load.
+
+---
+
+## How It Works
+
+During execution, the scale suite performs the following lifecycle:
+1. **Pre-flight process cleanup**: Detects and terminates any stale controller managers holding the configured metrics port (compatible with both Unix and Windows hosts).
+2. **Cluster Provisioning**: Automatically provisions a simulated KWOK cluster in the background using `kwokctl`.
+3. **Controller Startup**: Compiles the latest manager binary and starts it in the background targeting the KWOK cluster API server.
+4. **Scraper Registry**: Registers a metrics scraper targeting the controller's `/metrics` endpoint and spins up a local Prometheus scraper instance on the configured port with a `1s` scrape interval.
+5. **Phase Execution**:
+    *   **Tainting (Add) Phase**: Registers the `NodeReadinessRule` and triggers status transitions to `false`, measuring how fast the controller applies taints across all nodes.
+    *   **Untainting Phase**: Simulates status transitions to `true` (ready), measuring how fast the controller removes taints across all nodes.
+6. **Telemetry Report**: Queries the active Prometheus instance (calculating reconcile latencies, workqueue durations, custom node readiness counters, CPU rates, and memory peaks) and writes the results to the artifacts directory using Go's `text/template` engine.
+7. **Clean Teardown**: Gracefully deletes the KWOK cluster, stops Prometheus, and cleans up the controller manager process (unless `SKIP_TEARDOWN=true` is set).
+
+---
+
+## Configuration Parameters
+
+All scale parameters are configurable via shell environment variables passed directly to `make test-scale`.
+
+| Variable | Default | Description |
+| :--- | :--- | :--- |
+| `KWOKCTL_VERSION` | `v0.8.0` | Target KWOK cluster simulator release version to download and execute. |
+| `NODE_COUNT` | `1000` | The total number of simulated nodes to register. |
+| `NODE_CONCURRENT_RECONCILES` | `1` | Number of concurrent worker threads allocated to the Node controller. |
+| `RULE_CONCURRENT_RECONCILES` | `1` | Number of concurrent worker threads allocated to the NodeReadinessRule controller. |
+| `KUBE_API_QPS` | `-1` | The QPS limit for the controller's Kubernetes API client (disabled by default). |
+| `KUBE_API_BURST` | `-1` | The Burst limit for the controller's Kubernetes API client (disabled by default). |
+| `DISABLE_QPS_LIMITS` | `false` | Disable API Server write rate throttling inside the KWOK cluster. |
+| `NODE_LEASE_DURATION_SECONDS` | `40` | The duration of the node lease lease window configured in KWOK. |
+| `CONTROLLER_METRICS_PORT` | `8080` | Port for the controller's metrics endpoint. |
+| `PROMETHEUS_PORT` | `9090` | Port for the local Prometheus instance. |
+| `ARTIFACTS` | `test/scale/artifacts/` | Directory where test logs and reports are saved (populated by Prow for SpyGlass). |
+| `KUBECONFIG` | *Automatic* | Target kubeconfig path. Defaults to the managed KWOK cluster configuration. |
+| `SKIP_TEARDOWN` | `false` | If set to `true`, keeps the simulated nodes active, and the KWOK cluster/Prometheus instance running after the test finishes. |
+| `ENFORCEMENT_MODE` | `continuous` | Enforcement mode configured for the test `NodeReadinessRule` (e.g. `continuous`, `bootstrap-only`). |
+| `KWOK_RUNTIME` | `binary` | Cluster runtime mode configured for `kwokctl` (e.g. `binary`, `kind`). |
+
+---
+
+## Usage Examples
+
+Run all commands from the root directory of the repository.
+
+### 1. Default Scale Test (1000 Nodes)
+Runs the validation test with default QPS/concurrency settings:
+```bash
+make test-scale
+```
+
+### 2. Custom Validation Test (100 Nodes, 10 Workers)
+```bash
+NODE_COUNT=100 NODE_CONCURRENT_RECONCILES=10 make test-scale
+```
+
+### 3. Tuned High Scale Test (1000 Nodes)
+Runs a tuned high-scale configuration with increased concurrency and API server QPS limits:
+```bash
+NODE_COUNT=1000 NODE_CONCURRENT_RECONCILES=50 KUBE_API_QPS=500 KUBE_API_BURST=1000 DISABLE_QPS_LIMITS=true NODE_LEASE_DURATION_SECONDS=400 make test-scale
+```
+
+---
+
+## Output Artifacts
+
+Upon a successful test run, output files are populated in the configured `ARTIFACTS` directory:
+
+*   **`scalability_report.md`**: A detailed Markdown report capturing phase durations, P50/P90/P99 latency histograms, workqueue metrics, custom controller metrics (taint operations, rule evaluation durations, condition failures), and CPU/memory peaks.
+*   **`scalability_report.json`**: A structured JSON export of all scale test metrics grouped into categorized primitive types (`latencies`, `resources`, `workqueue`, `operations`) for automated processing and CI benchmarks.
+*   **`controller.log`**: Standard output logs generated by the controller manager daemon during execution.
+
+## Further Exploration
+
+By using `SKIP_TEARDOWN=true`, it is possible to access Prometheus's Web Dashboard at `localhost:9090` (or specified port) and query the database using PromQL.
+
+> [!IMPORTANT]
+> Skipping teardown requires deleting the KWOK cluster and finishing the controller process manually.

test/scale/promqueries.go

@@ -0,0 +1,188 @@
+//go:build scale
+
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scale
+
+// MetricQuery defines a query format for scraping Prometheus during scale tests.
+type MetricQuery struct {
+	Key       string
+	QueryTmpl string
+	Unit      string
+	IsCounter bool
+}
+
+// metricQueries contains the definitions of all Prometheus queries used in the scale tests.
+// Queries added here should also be added to the report template.
+var metricQueries = []MetricQuery{
+	{
+		Key:       "reconcile_time_p50",
+		QueryTmpl: "histogram_quantile(0.50, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "reconcile_time_p90",
+		QueryTmpl: "histogram_quantile(0.90, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "reconcile_time_p99",
+		QueryTmpl: "histogram_quantile(0.99, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "reconciliation_latency_p50",
+		QueryTmpl: "histogram_quantile(0.50, sum(rate(node_readiness_reconciliation_latency_seconds_bucket{rule=\"security-agent-readiness-rule\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "reconciliation_latency_p90",
+		QueryTmpl: "histogram_quantile(0.90, sum(rate(node_readiness_reconciliation_latency_seconds_bucket{rule=\"security-agent-readiness-rule\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "reconciliation_latency_p99",
+		QueryTmpl: "histogram_quantile(0.99, sum(rate(node_readiness_reconciliation_latency_seconds_bucket{rule=\"security-agent-readiness-rule\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+
+	{
+		Key:       "workqueue_queue_duration_p50",
+		QueryTmpl: "histogram_quantile(0.50, sum(rate(workqueue_queue_duration_seconds_bucket{name=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "workqueue_queue_duration_p90",
+		QueryTmpl: "histogram_quantile(0.90, sum(rate(workqueue_queue_duration_seconds_bucket{name=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "workqueue_queue_duration_p99",
+		QueryTmpl: "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{name=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "workqueue_work_duration_p50",
+		QueryTmpl: "histogram_quantile(0.50, sum(rate(workqueue_work_duration_seconds_bucket{name=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "workqueue_work_duration_p90",
+		QueryTmpl: "histogram_quantile(0.90, sum(rate(workqueue_work_duration_seconds_bucket{name=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "workqueue_work_duration_p99",
+		QueryTmpl: "histogram_quantile(0.99, sum(rate(workqueue_work_duration_seconds_bucket{name=\"node\",job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "rule_evaluation_duration_p50",
+		QueryTmpl: "histogram_quantile(0.50, sum(rate(node_readiness_evaluation_duration_seconds_bucket{job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "rule_evaluation_duration_p90",
+		QueryTmpl: "histogram_quantile(0.90, sum(rate(node_readiness_evaluation_duration_seconds_bucket{job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "rule_evaluation_duration_p99",
+		QueryTmpl: "histogram_quantile(0.99, sum(rate(node_readiness_evaluation_duration_seconds_bucket{job=\"node-readiness-controller\"}[%ds])) by (le))",
+		Unit:      "s",
+	},
+	{
+		Key:       "taint_operations_add",
+		QueryTmpl: "sum(node_readiness_taint_operations_total{rule=\"security-agent-readiness-rule\",operation=\"add\"}) - (sum(node_readiness_taint_operations_total{rule=\"security-agent-readiness-rule\",operation=\"add\"} @ %.3f) or vector(0))",
+		Unit:      "ops",
+		IsCounter: true,
+	},
+	{
+		Key:       "taint_operations_remove",
+		QueryTmpl: "sum(node_readiness_taint_operations_total{rule=\"security-agent-readiness-rule\",operation=\"remove\"}) - (sum(node_readiness_taint_operations_total{rule=\"security-agent-readiness-rule\",operation=\"remove\"} @ %.3f) or vector(0))",
+		Unit:      "ops",
+		IsCounter: true,
+	},
+
+	{
+		Key:       "condition_failures_total",
+		QueryTmpl: "sum(node_readiness_condition_failures_total{rule=\"security-agent-readiness-rule\"}) - (sum(node_readiness_condition_failures_total{rule=\"security-agent-readiness-rule\"} @ %.3f) or vector(0))",
+		Unit:      "failures",
+		IsCounter: true,
+	},
+	{
+		Key:       "operational_failures_total",
+		QueryTmpl: "sum(node_readiness_failures_total{rule=\"security-agent-readiness-rule\"}) - (sum(node_readiness_failures_total{rule=\"security-agent-readiness-rule\"} @ %.3f) or vector(0))",
+		Unit:      "failures",
+		IsCounter: true,
+	},
+	{
+		Key:       "cpu_cores_rate",
+		QueryTmpl: "sum(rate(process_cpu_seconds_total{job=\"node-readiness-controller\"}[%ds]))",
+		Unit:      "cores",
+	},
+	{
+		Key:       "cpu_cores_peak",
+		QueryTmpl: "max_over_time(sum(rate(process_cpu_seconds_total{job=\"node-readiness-controller\"}[5s]))[%ds:1s])",
+		Unit:      "cores",
+	},
+	{
+		Key:       "resident_memory_peak",
+		QueryTmpl: "max(max_over_time(process_resident_memory_bytes{job=\"node-readiness-controller\"}[%ds]))",
+		Unit:      "bytes",
+	},
+	{
+		Key:       "resident_memory_avg",
+		QueryTmpl: "avg(avg_over_time(process_resident_memory_bytes{job=\"node-readiness-controller\"}[%ds]))",
+		Unit:      "bytes",
+	},
+	{
+		Key:       "workqueue_retries_node",
+		QueryTmpl: "sum(workqueue_retries_total{name=\"node\",job=\"node-readiness-controller\"}) - (sum(workqueue_retries_total{name=\"node\",job=\"node-readiness-controller\"} @ %.3f) or vector(0))",
+		Unit:      "retries",
+		IsCounter: true,
+	},
+	{
+		Key:       "workqueue_retries_rules",
+		QueryTmpl: "sum(workqueue_retries_total{name=\"nodereadiness-controller\",job=\"node-readiness-controller\"}) - (sum(workqueue_retries_total{name=\"nodereadiness-controller\",job=\"node-readiness-controller\"} @ %.3f) or vector(0))",
+		Unit:      "retries",
+		IsCounter: true,
+	},
+	{
+		Key:       "workqueue_adds_node",
+		QueryTmpl: "sum(workqueue_adds_total{name=\"node\",job=\"node-readiness-controller\"}) - (sum(workqueue_adds_total{name=\"node\",job=\"node-readiness-controller\"} @ %.3f) or vector(0))",
+		Unit:      "adds",
+		IsCounter: true,
+	},
+	{
+		Key:       "workqueue_adds_rules",
+		QueryTmpl: "sum(workqueue_adds_total{name=\"nodereadiness-controller\",job=\"node-readiness-controller\"}) - (sum(workqueue_adds_total{name=\"nodereadiness-controller\",job=\"node-readiness-controller\"} @ %.3f) or vector(0))",
+		Unit:      "adds",
+		IsCounter: true,
+	},
+	{
+		Key:       "kube_api_requests_total",
+		QueryTmpl: "sum(rest_client_requests_total{job=\"node-readiness-controller\"}) - (sum(rest_client_requests_total{job=\"node-readiness-controller\"} @ %.3f) or vector(0))",
+		Unit:      "requests",
+		IsCounter: true,
+	},
+	{
+		Key:       "kube_api_requests_rate",
+		QueryTmpl: "sum(rate(rest_client_requests_total{job=\"node-readiness-controller\"}[%ds]))",
+		Unit:      "req/s",
+	},
+}

test/scale/report_types.go

@@ -0,0 +1,95 @@
+//go:build scale
+
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scale
+
+type queryResult struct {
+	Phase           string             `json:"phase"`
+	PhaseTitle      string             `json:"phase_title"`
+	DurationSeconds float64            `json:"duration_seconds"`
+	Metrics         map[string]string  `json:"metrics"`
+	RawMetrics      map[string]float64 `json:"raw_metrics"`
+}
+
+type ScalabilityReportJSON struct {
+	NodeCount int         `json:"node_count"`
+	Mode      string      `json:"mode"`
+	Runtime   string      `json:"runtime"`
+	Phases    []PhaseJSON `json:"phases"`
+}
+
+type PhaseJSON struct {
+	Phase           string         `json:"phase"`
+	DurationSeconds float64        `json:"duration_seconds"`
+	Latencies       LatenciesJSON  `json:"latencies"`
+	Resources       ResourcesJSON  `json:"resources"`
+	Workqueue       WorkqueueJSON  `json:"workqueue"`
+	Operations      OperationsJSON `json:"operations"`
+	APIClient       APIClientJSON  `json:"api_client"`
+}
+
+type PercentilesJSON struct {
+	P50 float64 `json:"p50"`
+	P90 float64 `json:"p90"`
+	P99 float64 `json:"p99"`
+}
+
+type LatenciesJSON struct {
+	ReconcileTime          PercentilesJSON `json:"reconcile_time"`
+	ReconciliationLatency  PercentilesJSON `json:"reconciliation_latency"`
+	RuleEvaluationDuration PercentilesJSON `json:"rule_evaluation_duration"`
+	WorkqueueQueueDuration PercentilesJSON `json:"workqueue_queue_duration"`
+	WorkqueueWorkDuration  PercentilesJSON `json:"workqueue_work_duration"`
+}
+
+type CPUUsageJSON struct {
+	Rate float64 `json:"rate"`
+	Peak float64 `json:"peak"`
+}
+
+type MemoryUsageJSON struct {
+	AvgBytes  float64 `json:"avg_bytes"`
+	PeakBytes float64 `json:"peak_bytes"`
+}
+
+type ResourcesJSON struct {
+	CPUCores       CPUUsageJSON    `json:"cpu_cores"`
+	ResidentMemory MemoryUsageJSON `json:"resident_memory"`
+}
+
+type ControllerWorkqueueJSON struct {
+	Adds    int64 `json:"adds"`
+	Retries int64 `json:"retries"`
+}
+
+type WorkqueueJSON struct {
+	Node  ControllerWorkqueueJSON `json:"node"`
+	Rules ControllerWorkqueueJSON `json:"rules"`
+}
+
+type OperationsJSON struct {
+	TaintsAdded         int64 `json:"taints_added"`
+	TaintsRemoved       int64 `json:"taints_removed"`
+	ConditionFailures   int64 `json:"condition_failures"`
+	OperationalFailures int64 `json:"operational_failures"`
+}
+
+type APIClientJSON struct {
+	RequestsTotal int64   `json:"requests_total"`
+	RequestsRate  float64 `json:"requests_rate"`
+}