A Gateway that never becomes ready is carried forever with nobody told

[ecv]

Summary

A Gateway that a controller accepts and then never programs stays that way forever, and nobody is told.

Ten Gateways in production have sat in that state for between four and eight months, the oldest created in January and the newest in May. Every one of them reached Accepted and never reached Programmed, so a user who created a Gateway and walked away is left holding an object that is permanently half made.

The only thing that notices is an operator alert about somebody else's project, which no Datum engineer can act on.

What should happen

A Gateway that has not reached both Accepted and Programmed within a bounded window should either be reaped or reported to the person who owns it. Which of those two, and how long the window runs, is the product decision this issue is asking for.

The current behaviour has no window at all. Nothing expires, nothing escalates, and the object's owner never learns that the thing they created does not work.

What success looks like

• A Gateway that never becomes ready inside a bounded window either goes away or its owner hears about it
• The window, and the choice between reaping and notifying, are configurable and written down
• Clearing the backlog stops depending on an operator spotting an alert about a project they cannot touch

Context

The alert that surfaced this is described in Datum's deployment runbook, which also carries the queries used to produce the counts above.

https://github.com/datum-cloud/infra/issues/4909

https://github.com/datum-cloud/infra/blob/main/docs/runbooks/gateway-never-programmed.md

[ecv]

Evidence for the window this asks for, from checking where the ten actually resolve.

Seven are abandonment. Four hostnames have no DNS record at all and three point at other providers, so those users created a Gateway, never finished, and left. A bounded window would have reaped or notified all seven long ago.

Three point at Datum and one of those is a customer-visible outage with an incident open since January, https://github.com/datum-cloud/engineering/issues/141, still reproducible today. So the window cannot be a reaper alone. A Gateway that never programs while traffic is arriving at it needs escalating, not removing.

That split is the argument for making the choice configurable rather than picking one: the same unprogrammed state means "user walked away" in seven cases and "we are failing a live request" in two.

The measurements are in https://github.com/datum-cloud/infra/issues/4967.