Report Remote Configuration and workload autoscaling in the Cluster Agent status - #55619
Conversation
There was a problem hiding this comment.
AI review by Codex (OpenAI) - workflow run
Patch is incorrect: workload autoscaling status can retain stale version and error information after newer Remote Configuration updates.
81e329d to
62e66bf
Compare
There was a problem hiding this comment.
AI review by Codex (OpenAI) - workflow run
patch is correct — the status reporting, synchronization, rendering, and tests are consistent with the intended behavior.
There was a problem hiding this comment.
More details
The static review found no reportable defect. The status output handles disabled, not-started, and active workload autoscaling states, and it lists extra Remote Configuration clients.
🤖 Datadog Autotest · Commit 62e66bf · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
🎯 Code Coverage (details) 🔗 Commit SHA: c83527a | Docs | View more details | Give us feedback! |
The template ended with `{{- end -}}`, whose trailing `-}}` swallowed the
file's last newline, so the section emitted none at all. The status renderer
appends a single newline after each section and writes the next section's rule
immediately, so whatever follows secrets lands flush against its header.
This was latent because secrets was the last section rendered. Every other
status template already ends with a newline.
Assisted-by: Claude:claude-opus-5
Per-instance Remote Configuration status is exported but only the default client was rendered, so a Cluster Agent with additional clients gave no sign they existed and a failing one was invisible to `status`. List every instance other than the default under an "Additional clients" heading, with the endpoint it polls, which is the point of having them: they can target a different backend. The default client is filtered out by the provider rather than by name inside the templates, so the heading appears only when there is something to show. Assisted-by: Claude:claude-opus-5
There was no way to tell from `datadog-cluster-agent status` whether workload autoscaling was receiving anything over Remote Configuration. Report the number of DatadogPodAutoscalers, leader state, which Remote Configuration client serves the products, and per product the last config version received, when it arrived, how many configs it carried and the last apply error. The update is recorded before the configs are processed, so one that fails to apply still shows as received and its error is reported separately rather than looking like a connectivity problem. Three states are distinguished so the section is never ambiguous: disabled in configuration, enabled but not started, and running. The section is named "Autoscaling" because sections render in alphabetical order with only "collector" special cased, which places it after "Autodiscovery" and leaves room for cluster autoscaling to join it later. Only the workload products are reported for now. Assisted-by: Claude:claude-opus-5
An update carries the full current config set for a product, not a delta, so the version and error recorded for it describe that update alone. Both were accumulating instead: the version was a running maximum, so a config that was removed or replaced by a lower-version one left a version reported as current that no longer applied, and an apply error was never cleared, so a single transient failure stayed in the status for the life of the process even after the product recovered. Reset both at the start of each update. The cumulative update counter still accumulates, since that is what it is for. Also stop importing pkg/config/remote/service from the status provider. It was imported only to share the default instance name, and it drags uptane and bbolt into every binary that links the provider, which showed up as six new packages in process-agent. Repeating one string is much cheaper. Assisted-by: Claude:claude-opus-5
Renaming the sample instances shortened the keys, so gofmt wanted the values realigned. Caught by the linter, not by the tests. Assisted-by: Claude:claude-opus-5
c1d7c31 to
c83527a
Compare
Files inventory check summaryFile checks results against ancestor eb7754dc: Results for datadog-agent_7.84.0~devel.git.596.c83527a.pipeline.134080488-1_amd64.deb:No change detected Results for datadog-iot-agent_7.84.0~devel.git.596.c83527a.pipeline.134080488-1_amd64.deb:No change detected |
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: eb7754d Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_security_idle | memory utilization | +0.08 | [+0.03, +0.13] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -0.02 | [-0.07, +0.02] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.03 | [-0.07, +0.00] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_private_action_runner | memory utilization | -0.07 | [-0.19, +0.06] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | -0.12 | [-0.15, -0.08] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | -0.41 | [-0.49, -0.32] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_cpu | % cpu utilization | -0.65 | [-0.89, -0.40] | 1 | Logs |
| ➖ | quality_gate_metrics_logs | memory utilization | -0.86 | [-1.10, -0.62] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_memory | memory utilization | -0.96 | [-1.17, -0.75] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | -2.29 | [-3.15, -1.44] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 172.78MiB ≤ 179MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 749.69KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 515.85MiB ≤ 537MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.13MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 20 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 213.26MiB ≤ 220MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 263.51MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 417.36 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 20 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 419.43MiB ≤ 455MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.94GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 72.06MiB ≤ 75MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 28.89 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 326.99MiB ≤ 355MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 62.13 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 301.06MiB ≤ 335MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 22.54 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 311.04MiB ≤ 345MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
What does this PR do?
Reports Remote Configuration and workload autoscaling health in
datadog-cluster-agent status.Every Remote Configuration client is listed, with the endpoint it polls:
And a new
Autoscalingsection reports what workload autoscaling is receiving:Both samples are taken from a Cluster Agent whose autoscaling client targets a different Remote Configuration backend than its default client, with the extra endpoint anonymized.
Motivation
Per-instance Remote Configuration status was exported but never rendered, so a Cluster Agent with additional clients gave no sign they existed and a failing one was invisible to
status.Separately, there was no way to tell whether workload autoscaling was receiving anything over Remote Configuration, which is the first question asked when an autoscaler is not behaving.
Describe how you validated your changes
Unit tests over the text, JSON and HTML renderings, including an unauthorized extra client, the default client not being listed twice, and the three workload autoscaling states (disabled, enabled but not started, running).
Also validated end to end on a local kind cluster: both sections rendered as shown above against live configuration arriving from a second backend.
Additional Notes
Also fixes a missing trailing newline in the secrets status template. It ended with
{{- end -}}, which swallowed the file's last newline, so any section rendered after it landed flush against its header. This was latent because secrets was previously the last section.Stacked on #55618, which adds the additional clients this reports on.
🤖 PR description and code assisted by Claude:claude-opus-5