Skip to content

Gateways with an unverified hostname rewrite status once a second #404

Description

@scotwells

Summary

A Gateway listener with an unverified hostname is dropped. The operator sets Programmed to false, skips the downstream write, and requeues immediately. The status it writes is already correct, so nothing changes and the requeue repeats. There is no backoff.

Impact

Nine production gateways write status about once per second, roughly 966,000 writes per day. Each write is stored at RequestResponse with both bodies and triggers an API server admission read. This is about a fifth of all platform audit events and more than half the bytes stored.

Evidence

Over seven days: 3.4 million Gateway status updates against 792 reads. The affected gateways hold the same resourceVersion, generation and status hash across the entire window. One gateway took 53 consecutive updates in 92 seconds under a single resourceVersion.

The trigger is a listener reporting UnverifiedHostnamesPresent. That condition clears only when the domain is verified, so the retry has nothing to converge on.

Introduced in v0.26.0. Still present in v0.26.1.

Success looks like

  • Status is not written when the computed status matches the observed status
  • A condition that cannot resolve without user action backs off instead of retrying at reconcile speed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions