[mimir-distributed] Helm chart: no way to make store-gateway cache reads zone-local → large inter-zone egress cost

[SLoeuillet]

[mimir-distributed] Helm chart: no way to make store-gateway cache reads zone-local → large inter-zone egress cost

Chart: mimir-distributed 6.1.0 (Mimir 3.1.2), Helm-based deploy on GKE (3 AZs)

The problem: inter-zone egress cost

We run zone-aware store-gateways (store-gateway-zone-a/b/c) but the chart deploys the block-store memcached caches (chunks-cache, index-cache, …) as a single shared cluster. The memcached client consistent-hashes keys across all cache pods with no zone awareness, so each store-gateway reads roughly (N-1)/N of its chunks from a pod in another availability zone.

On our cluster this is by far the dominant inter-zone talker. Measured from the chunks-cache memcached exporters:

• ~600 MiB/s served, evenly split across the 3 cache pods → ~146 TiB / 3 days, ~⅔ of it cross-zone (confirmed against VPC Flow Logs).
• hit ratio 99.999%, working set only ~0.7 GiB — so this is almost entirely hot cache hits being shipped across zones, i.e. pure inter-AZ network cost for data that is already cached in every zone's reach.

Same-region inter-AZ traffic is billed per GiB on the major clouds, so this is a real and ongoing cost, not just a latency concern.

The zone-aware replication docs already acknowledge this:

Most cloud providers charge for inter-availability zone networking. Deploying Grafana Mimir with zone-aware replication across multiple cloud provider availability zones likely results in additional networking costs.

…but that guidance only covers alertmanager, ingesters and store-gateways. The block-store memcached caches are never mentioned, even though — as the numbers above show — the store-gateway↔chunks-cache read path is the single largest inter-AZ contributor in our deployment. So the caches are a documented-cost blind spot: zone awareness is offered for the components the docs warn about, but not for the one that actually dominates the bill.

Why the usual levers don't apply

• Spreading the cache 1-per-zone doesn't help. Consistent hashing means a given key lives on exactly one pod, so each store-gateway still sends ~⅔ of reads cross-zone regardless of pod placement.
• Kubernetes Topology Aware Routing / trafficDistribution: PreferClose don't apply. The cache Service is headless and the memcached client resolves it via DNS SRV and dials pod IPs directly, bypassing kube-proxy. And "prefer a local endpoint" is semantically wrong for a hash-sharded cache — a key that hashes to the zone-a pod served from the zone-b pod is just a miss.
• The Jsonnet libs already solve this (Jsonnet: add multi-zone read path components support #13559 added multi-zone-memcached.libsonnet with per-zone cache instances + zonal addresses + a memberlist bridge), but there is no equivalent in the mimir-distributed Helm chart — the store-gateway config is rendered once and shared across all zones, and the cache address is a single global value.

How we worked around it (the ugly way)

Since we run exactly one store-gateway per zone, we retired the shared cache (chunks-cache.replicas: 0) and gave each store-gateway its own memcached sidecar, pointing the store-gateway at localhost:

store_gateway:
  extraArgs:
    blocks-storage.bucket-store.chunks-cache.memcached.addresses: "dns+localhost:11211"
  extraContainers:
    - name: chunks-cache            # memcached sidecar sized to the working set
      image: memcached:1.6.34-alpine
      args: ["-m 2048", "-I 1m", "-c 16384", "-u 11211"]
      ports: [{ containerPort: 11211, name: client }]
    - name: chunks-cache-exporter   # + a Service extraPort + ServiceMonitor to keep metrics
      image: prom/memcached-exporter:v0.15.3
      args: ["--memcached.address=localhost:11211", "--web.listen-address=0.0.0.0:9150"]

This drops cross-zone chunk reads to zero. But it's a hack: it only works cleanly at 1 store-gateway/zone (a sidecar is per-pod, not per-zone), it hand-rolls the memcached + exporter + a custom ServiceMonitor that the chart would otherwise manage, and it silently diverges the store-gateway config from the shared cache config.

Proposed fix: extend the Helm chart a bit

The store-gateway StatefulSet template already loops per zone with $zoneName in scope and already emits a per-zone flag (-store-gateway.sharding-ring.instance-availability-zone={{ $zoneName }}). Two options, smallest first:

1. Per-zone extraArgs on the store-gateway zone struct. Add an optional extraArgs field alongside the existing nodeSelector/extraAffinity/storageClass per zone, merged after the shared store_gateway.extraArgs. That alone lets operators point each zone's store-gateway at a zone-local cache address — a few lines in the existing loop.

2. First-class zone-aware caches (mirrors Jsonnet: add multi-zone read path components support #13559 and the chart's existing zoneAwareReplication): a chunks-cache.zoneAwareReplication.enabled (and index-cache) that renders one cache StatefulSet per zone with zone affinity, and auto-wires each store-gateway-zone-X to its zone-local cache addresses. This matches how ingester/store-gateway zone awareness already works in the chart and would make the Jsonnet capability available to Helm users.

Either would remove the need for the sidecar hack and let Helm-based, zone-aware deployments avoid the inter-zone cache egress. Happy to help / open a PR if there's interest and a preferred direction.