DNSRecordSet delete can orphan the PowerDNS RRset: no finalizer on PDNS controller + dead cleanup path

[ecv]

Summary

Removing a DNSRecordSet does not reliably remove the record from PowerDNS. If the operator process restarts during the delete window, the PowerDNS RRset is orphaned — nothing re-triggers its cleanup — so a stale CNAME/ALIAS/A/etc. keeps resolving after the owning resource is gone.

The delete logic, while the operator is running, is correct: it targets the exact (zone, type, name) RRset and retries through API errors (a 422 does not drop anything). The gap is durability across a restart, plus a dead cleanup path that was never implemented.

Root cause

1. The PowerDNS-writing controller registers no finalizer. DNSRecordSetPowerDNSReconciler (internal/controller/dnsrecordset_powerdns_controller.go) has no finalizer at all. Deletion is driven only by an in-memory watch DeleteFunc + workqueue; the actual removal happens when reconcile sees no live CR owns the tuple:

   // dnsrecordset_powerdns_controller.go:126-127
   if owner == nil {
       pdnsErr = r.PDNS.DeleteRRSet(ctx, zone.Spec.DomainName, req.RecordSetType, req.RecordSetName)

2. The downstream finalizer removes itself without cleaning PowerDNS. downstreamRSFinalizer = "dns.networking.miloapis.com/finalize-dnsrecordset-downstream" (dnsrecordset_downstream_controller.go:33) is dropped unconditionally on delete, with no PDNS call:

   // dnsrecordset_downstream_controller.go:64-73
   if controllerutil.ContainsFinalizer(&rs, downstreamRSFinalizer) {
       base := rs.DeepCopy()
       controllerutil.RemoveFinalizer(&rs, downstreamRSFinalizer)   // no PDNS cleanup
       if err := r.Patch(ctx, &rs, client.MergeFrom(base)); err != nil { ... }
   }

3. The cleanup helper is a dead comment. cleanupPDNSForRecordSet is referenced only by a doc comment (dnsrecordset_downstream_controller.go:212) and is not defined anywhere in internal/.

So once the CR leaves etcd, the queued PDNS delete is the only thing standing. A crash/restart in that window loses it; zone-level re-enqueue only fans out to CRs that still exist, so nothing re-drives the orphaned RRset.

Impact

Orphaned provider records (CNAME/ALIAS especially) that survive proxy/hostname teardown — keep resolving to a canonical name that no longer routes, and can block a later resource from re-claiming the same name.

Fix direction

Give the record a finalizer that guarantees PDNS deletion before removal — either on the PowerDNS controller, or implement cleanupPDNSForRecordSet and gate downstreamRSFinalizer removal on its success (retry on error rather than removing unconditionally).

Related

• https://github.com/datum-cloud/infra/issues/3407 — real-world orphaned ALIAS (www.ab.dk) observed in prod
• https://github.com/datum-cloud/infra/issues/1770 — external-dns not cleaning orphaned TXT records on delete (same failure shape at the other DNS path)
• Stale hostname CNAME/ALIAS not reaped on hostname removal — breaks resolution (SERVFAIL) network-services-operator#283 — upstream DNSRecordSet lifecycle

Verified on main @ 81f7549.

[ecv]

Test coverage

No test would catch this. Happy-path A-record deletion is covered (fake-client unit at dnsrecordset_powerdns_controller_test.go:326,495 + one chainsaw e2e, test/e2e/zones-and-records), but in every case the operator stays running and the owner == nil condition is fabricated directly rather than driven by a real CR deletion — the informer DeleteFunc wiring (dnsrecordset_powerdns_controller.go:435) that this bug hinges on is untested.

Gaps to close alongside the fix:

• Restart / lost-delete durability: delete a DNSRecordSet, simulate an operator gap (or the CR removed before PDNS delete succeeds), assert the RRset is gone from PDNS. Zero coverage today.
• ALIAS delete: no test deletes an ALIAS DNSRecordSet and asserts the ALIAS RRset is removed (ALIAS create is covered; delete is not). DeleteRRSet is never exercised below the fake-controller layer.