dnsrecordset-powerdns queue saturates (p95 wait ~79s, depth ~1555): raise concurrency, cut churn, fix fresh-zone backoff

[ecv]

Summary

The dnsrecordset-powerdns controller cannot keep up with DNS record programming under load. Its workqueue in datum-dns-system shows p95 queue wait ≈ 78.8s and depth bursting to ≈1555, while per-reconcile compute stays <250ms. Records therefore take 60–85s to converge — the queue wait, not PowerDNS or CPU, is the latency.

This is the shared root cause of two convergence-SLO breaches tracked upstream:

• Custom-hostname DNSRecordProgrammed stuck Pending >60s on fresh zone (breaches convergence SLO) network-services-operator#291 — custom-hostname DNSRecordProgrammed stuck Pending >60s on a fresh zone.
• Gateway *.datumproxy.net address resolves in 65–85s, exceeding the 60s convergence SLO network-services-operator#292 — Gateway *.datumproxy.net address resolves in 65–85s.

Both exceed the 60s managed-DNS-record programming threshold. Per-stage attribution is in datum-cloud/infra#3622 (stages 4/5). One fix here should clear both.

Impact

• Customer hostnames and gateway addresses are "programmed" but do not resolve for over a minute, failing the scheduled prod e2e suite once the budget was tightened to the 60s SLO.
• Chronic, load-dependent: worsens with the number of DNSRecordSets/records per zone and with event bursts (e.g. many HTTPProxies/gateways created together).

Root cause (internal/controller/dnsrecordset_powerdns_controller.go)

1. Worker cap = 4. MaxConcurrentReconciles defaults to 4 (internal/config/config.go:36, internal/config/zz_generated.defaults.go:27; consumed at lines 407, 421). The agent runs --leader-elect, so only one pod is active and all record programming drains through 4 goroutines doing serialized PowerDNS PATCHes (ReplaceRRSet→applyRRSetPatch, internal/pdns/client.go:335-405).
2. Per-record-name fan-out. Each request is a single (zone,type,name) tuple (lines 31-36). enqueueRecordSetRequests emits one item per record name (lines 458-479); the DNSZone watch re-enqueues every record of every DNSRecordSet in the zone on any zone event (enqueueZoneRecords, lines 481-500). One zone event → N items → depth spikes.
3. Self-inflicted churn. The DNSRecordSet UpdateFunc enqueues both new and old objects with no generation/status predicate (lines 431-433); every reconcile writes status (lines 226, 242), re-triggering the watch. This is the loop behind DNSZones and DNSRecordSets seem to be patching status too often #21.
4. Fresh-zone exponential backoff. On a zone not yet present in PowerDNS, ReplaceRRSet returns a non-2xx error; Reconcile returns it (lines 146-148), requeuing through the exponential failure limiter (line 417) with base 1s → max 30s (config.go:41,46). Backoff stacks 1→2→4→8→16→30s, pushing fresh-zone records past 60s (the network-services-operator#291 signature). There is no QPS/token-bucket limiter — only this failure limiter.

Proposed fix

dns-operator:

• Raise MaxConcurrentReconciles default 4 → ~32 (config.go:36; regenerate defaults). Primary lever, already plumbed.
• Add a watch predicate ignoring status-only/no-op updates (e.g. GenerationChangedPredicate) and stop enqueuing ObjectOld except when the record-name set changes (SetupWithManager, lines 427-443). Fixes DNSZones and DNSRecordSets seem to be patching status too often #21 and shrinks depth bursts.
• On zone-not-yet-in-PowerDNS, requeue with a short fixed RequeueAfter (~1–2s) instead of erroring into the exponential limiter (lines 146-148); optionally lower rateLimiterMaxDelay default 30s → ~5s.
• (Optional, larger) reconcile per (zone,type) and batch RRsets — applyRRSetPatch / ApplyRecordSetAuthoritative (client.go:272-332) already accept multi-RRset payloads.

Infra overlay (agent server-config ConfigMap, currently all-commented at config/agent/server-config.yaml:12-15):

controllers:
  dnsRecordSetPowerDNS:
    maxConcurrentReconciles: 32
    rateLimiterMaxDelay: 5s

Raising StatefulSet replicas does not help — leader election makes concurrency per-active-pod; the knob is maxConcurrentReconciles.

Verification

• workqueue_queue_duration_seconds{name="dnsrecordset-powerdns"} p95 drops from ~79s toward single-digit seconds; workqueue_depth{name="dnsrecordset-powerdns"} peak falls well below 1555 under the same load.
• Re-run the network-services-operator#291/#292 e2e loops: custom-hostname DNSRecordProgrammed=True and *.datumproxy.net resolution both land under 60s (target <30s) on a fresh zone.
• Confirm status-patch/reconcile rate for dnsrecordset-powerdns drops after the predicate change (validates DNSZones and DNSRecordSets seem to be patching status too often #21).

Related: #21 (status churn — subsumed), datum-cloud/network-services-operator#291, datum-cloud/network-services-operator#292, datum-cloud/infra#3622.