Customers who delegate to Datum cannot finish domain verification

[ecv]

Summary

A customer who proves ownership by pointing their registrar at Datum's nameservers cannot finish verification, and their zone never provisions.

Verification waits for a live zone while provisioning waits for verification, so each side holds the other.

The first attempt at a fix treated the delegation itself as the proof. It was reviewed and closed, because it could not execute against real objects and because the proof it rests on cannot say which customer owns the name.

Closing this needs a change in both operators rather than in this one alone.

How it fails

1. A Domain is created for a name that has never been verified.
2. The zone for it is held unprovisioned, pending domain verification.
3. Verification looks for a zone that is accepted and programmed, finds none, and never grants verified.
4. The two wait on each other indefinitely. The verification record value is a per-Domain identifier, so nothing can be pre-published to break the cycle.

The provisioning gate itself is correct and stays. Serving DNS for unverified domains is the hole it closes.

Why the first attempt failed

Both findings came out of an adversarial review of #422, which carries the detail.

• The cycle has an edge in dns-operator. Verification finds candidate zones through a status field that dns-operator writes only after the gate returns, so a zone held pending verification is invisible to the check meant to release it. Nothing in this repo reaches that ordering.
• Delegation to Datum is not proof of ownership. The nameserver list is one global set shared by every customer, and a name with no delegation of its own inherits its parent's, so one customer's delegation can verify a subdomain claimed by another. Verification also never re-runs, so a wrong answer is permanent.

What success looks like

• A customer who has delegated to Datum completes verification without the zone being programmed first
• Verification establishes which customer owns the name, so one customer's delegation cannot verify a name claimed by another, including a subdomain of it
• Verification is re-checked rather than latched, so a withdrawn delegation does not leave a permanent claim
• Both operators are exercised together by a test, which nothing does today
• Staging end to end DNS zones verify inside the test budget with the gate still in place
• The next stable dns-operator release is safe to ship to production

Evidence

• The deadlock reproduces on every staging end to end run since the gate reached staging, with zones pinned pending domain verification and the operator logging that it is deferring DNS provisioning
• Production runs a pre-gate release and verifies per-run subdomains in under a second, which is why the deadlock has not reached customers yet
• The verification check is the same on main and on the staging prototype branch, ruling out a second cause on this side

Warnings

Warning

Cutting a stable dns-operator release before this lands ships the deadlock to production customers.

Note

Staging is pinned to a prerelease filter for separate work, so a fix merged here does not reach staging and https://github.com/datum-cloud/infra/issues/4310 stays red until that pin is restored.

Related to https://github.com/datum-cloud/infra/issues/4310
Related to #290

[ecv]

Trip report on the attempt at this. #422 is closed do not merge, and the full review sits on that pull request. Short version of what it changes about this issue.

The approach in that branch was to accept registry delegation to Datum's nameservers as proof of ownership, so verification stops waiting on a live zone. Three independent adversarial reviews took it apart, and two of the findings are about this issue rather than about that diff.

First, the fix could not have executed. Verification finds candidate zones with a field selector on status.domainRef.name, and dns-operator writes status.domainRef only in updateStatus, which runs after the verification gate returns. A zone parked at PendingDomainVerification therefore has no domainRef, verification reports DNSZoneNotFound, and never reaches the delegation check at all. The deadlock in this issue has an edge in dns-operator that no change in this repo can reach. Whichever way verification is eventually proven, that ordering has to change first.

Second, delegation cannot be the proof. The class nameserver list is one global set shared by every tenant, so an overlap test shows that a name is delegated to Datum and never that this project owns it. Registry lookup for a name with no cut of its own falls back to the apex nameservers, so a second tenant can verify sub.victim.com off a victim's apex delegation without touching the victim's DNS. Verification is also a latch that nothing revokes, which makes any wrong answer permanent.

So this issue needs rescoping rather than another attempt at the same shape. The work is a dns-operator change to populate domainRef before the gate, a proof that carries tenant identity, either a per-tenant token in the delegation or a name the tenant has to place, and coverage that runs both operators together, which nothing does today.

Two loose ends found on the way, neither of them this issue. Staging NSO is pinned to a prerelease filter for another change, so a merge here reaches production and not staging, and https://github.com/datum-cloud/infra/issues/4310 would stay red on merge regardless. And the alert on zones held in PendingDomainVerification, which 4310 says should ship whatever happens to the fix, is still unfiled.