Skip to content

refactor(fqdn-store): bound the store with pingora-memory-cache#454

Merged
pigri merged 1 commit into
feat/dns-time-preblockfrom
refactor/fqdn-store-memory-cache
Jul 20, 2026
Merged

refactor(fqdn-store): bound the store with pingora-memory-cache#454
pigri merged 1 commit into
feat/dns-time-preblockfrom
refactor/fqdn-store-memory-cache

Conversation

@pigri

@pigri pigri commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Stacked on #453 — review/merge that first; this PR targets its branch so the diff stays to the store swap alone.

Problem

The DNS-snoop ip -> fqdn store is an unbounded DashMap kept in check by a dedicated thread doing a full retain sweep every 60s. Since the TTL floor is 60s, every binding lives at least a minute, so a host resolving at a high rate — or anything spraying lookups — grows the map freely between sweeps, with no ceiling. The sweep itself is an O(n) pass taking write locks across shards.

Change

Back it with pingora-memory-cache (S3-FIFO + TinyLFU), already a workspace dependency used by synapse-proxy / -core / -worker / -waf. That gives:

  • a hard capacity ceiling instead of unbounded growth
  • lazy expiry on read, so the sweeper thread and its periodic O(n) scan both disappear

Lookups stay TTL-respecting — deliberately get, not get_stale

Worth stating explicitly, since "prefer stale over a miss" looks appealing for a security lookup and is the wrong call here.

lookup() has always failed open on expiry, by design: the TTL is capped precisely "so stale bindings expire", because an address may be reassigned. Serving the old name after expiry would block whoever holds that address now — trading a known, bounded failure mode for collateral damage against an innocent host.

So expiry behaviour is unchanged. The only genuinely new failure mode is eviction under capacity pressure, which turns a live binding into a miss and silently leaves a dns.fqdn rule unmatched.

Mitigated two ways rather than by resurrecting expired data:

  • capacity sized well above a full TTL window of answers
  • lookups now counted by outcome — lookup_hits / lookup_expired / lookup_misses — so misses climbing against a steady record rate makes the pressure visible instead of silent

Tradeoff worth a reviewer's eye

MemoryCache keys by the hash of the address, not the address itself, so a 64-bit collision would mis-attribute one binding to another. Negligible at this capacity, and the same tradeoff the other pingora-backed caches in this repo already accept — but it is a real semantic difference from the DashMap it replaces.

Tests

  • lookup outcomes are counted correctly (hit vs miss)
  • the store stays usable under capacity pressure — a freshly recorded binding still resolves after the cache is pushed well past a quarter of its capacity
  • existing coverage (parse, record/lookup roundtrip, queries, hook dispatch) unchanged and still green

The store and its counters are process-global, so the tests that record or look up share a lock — without it the counter deltas race against other tests in the same binary (caught while writing them).

Verification

  • cargo +nightly fmt -- --check — clean
  • cargo clippy --locked --workspace --all-targets -- --deny warnings — clean
  • cargo clippy --locked --workspace --features classifier --all-targets -- --deny warnings — clean
  • cargo test --locked --workspace --lib — all 19 binaries green
  • fqdn_store suite run repeatedly to confirm no flakiness; feat(dns): decide dns.fqdn rules when the answer is snooped #453's dns_preblock + dns_fqdn_repro still pass on the new backend
  • Cargo.lock moves by exactly one line (the new dependency edge)

Unrelated, noted in passing: synapse-core's service_graph::tests::buffer_behaviour failed once under parallel execution and then passed 5/5 in isolation and 4/4 in full runs. It is independent of this change (synapse-core depends on none of the touched crates), but it is an intermittent flake sitting in the wellness gate.

The DNS-snoop `ip -> fqdn` store was an unbounded DashMap kept in check by a
dedicated thread doing a full `retain` sweep every 60s. Because the TTL floor is
60s, every binding lives at least a minute, so a host resolving at a high rate —
or anything spraying lookups — grew the map freely between sweeps with no
ceiling.

Back it with pingora-memory-cache (S3-FIFO + TinyLFU), already a workspace
dependency. That caps live bindings at a fixed capacity and expires lazily on
read, so the sweeper thread and its periodic O(n) scan both go away.

Lookups stay TTL-respecting (`get`, not `get_stale`). A binding past its TTL is
still treated as absent: the address may have been reassigned since, and serving
the old name would block whoever holds it now — the reason the TTL is capped in
the first place. Behaviour on expiry is therefore unchanged.

The one new failure mode is eviction under capacity pressure, which turns a live
binding into a miss and silently leaves a `dns.fqdn` rule unmatched. Capacity is
sized well above a full TTL window of answers, and lookups are now counted by
outcome (hit / expired / miss) so the pressure is visible rather than silent.

Entries are keyed by the hash of the address rather than the address itself, so
a 64-bit collision would mis-attribute a binding — negligible at this capacity,
and the same tradeoff the other pingora-backed caches here already accept.

Tests covering the counters and behaviour under capacity pressure share a lock,
since the store and its counters are process-global and would otherwise race.
@pigri
pigri merged commit 5690b77 into feat/dns-time-preblock Jul 20, 2026
13 checks passed
@pigri
pigri deleted the refactor/fqdn-store-memory-cache branch July 20, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant