Summary
tabletAvailableSeconds is hardcoded to 30s in reconcile_tablets.go, but vtgate's default --tablet_refresh_interval is 1m. When a tablet restarts and gets a new IP, vtgates rediscover it by polling the topology — this can take up to the full tablet_refresh_interval. Because the operator considers a tablet Available after only 30s, it proceeds to drain the next replica before most vtgates have discovered the first one, causing "no healthy tablet available for tablet_type:REPLICA" errors during rolling restarts.
We would like tabletAvailableSeconds to be exposed as a configurable field on the VitessCluster / VitessShard CRD so operators can align it with their vtgate topology poll interval without forking the operator or tuning vtgate flags.
Root Cause
The timing contract that tabletAvailableSeconds enforces
tabletAvailableSeconds is meant to create a safety buffer: the operator only calls a tablet Available — and therefore safe to trigger the next rolling step — after it has been continuously Ready for at least that many seconds. The assumption is that vtgates will have finished discovering the new tablet endpoint within that window.
Why the default breaks that contract
vtgate polls the topology for tablet health on the interval set by --tablet_refresh_interval (default 1 minute). The poll is not edge-triggered. A tablet that becomes Ready just after a poll will not be discovered by vtgate until the next poll — up to tablet_refresh_interval later.
With tabletAvailableSeconds = 30s and tablet_refresh_interval = 60s (default):
t=0s Replica A restarts, pod becomes Ready, gets a new IP
t=1s Most vtgates polled just before t=0 — they do not know about A yet
t=30s Operator marks A as Available; begins draining Replica B
t=31s Replica B goes NOT_SERVING
→ vtgates that have not yet discovered A still only know about B (now NOT_SERVING)
→ those vtgates have zero healthy REPLICA tablets
→ "no healthy tablet available for 'tablet_type:REPLICA'"
t=51s First vtgates discover A is healthy — errors stop for them
t=60s Last vtgates discover A — all vtgates recover
The exposure window = actual discovery lag − tabletAvailableSeconds. We observed discovery lags of 45–51 seconds in production, leaving a window of up to 21 seconds where errors occur.
Real Incident
Observed vtgate discovery lag
After Replica A restarted and began serving at t=0, vtgates discovered it over a 51-second window across 10 vtgate pods:
[t+0s] vtgate-01: replica-A false => true (REPLICA) ← 1st: immediate
[t+44s] vtgate-02: replica-A false => true (REPLICA) ← 2nd: 44s later
[t+46s] vtgate-03: replica-A false => true (REPLICA)
[t+46s] vtgate-04: replica-A false => true (REPLICA)
[t+46s] vtgate-05: replica-A false => true (REPLICA)
[t+46s] vtgate-06: replica-A false => true (REPLICA)
[t+46s] vtgate-07: replica-A false => true (REPLICA)
[t+46s] vtgate-08: replica-A false => true (REPLICA)
[t+48s] vtgate-09: replica-A false => true (REPLICA)
[t+51s] vtgate-10: replica-A false => true (REPLICA) ← last: 51s later
Only 1 of 10 vtgates discovered the restarted replica immediately. The remaining 9 took 44–51 seconds — consistent with --tablet_refresh_interval=1m.
How the errors occurred (two bursts)
Burst 1 (~24 errors over ~6s):
The operator considered Replica A Available ~30s after it started serving. Shortly after, the operator scheduled a drain on Replica B. When Replica B went NOT_SERVING (MySQL socket gone during pod restart), 9 of 10 vtgates had still not discovered Replica A — they saw zero REPLICA tablets.
Replica B: true => false (REPLICA)
error: dial unix /vt/socket/mysql.sock: no such file or directory
→ 9/10 vtgates have zero healthy REPLICA tablets
→ "no healthy tablet available for 'tablet_type:REPLICA'"
Burst 2 (~14 errors over ~7s):
After PRS, the former primary briefly became a REPLICA and then its pod was restarted. Two vtgates that had not yet discovered Replica B as serving again had zero REPLICA tablets.
Former primary: true => false (REPLICA)
error: dial unix /vt/socket/mysql.sock: no such file or directory
vtgate-09: no healthy tablet available for 'tablet_type:REPLICA'
vtgate-10: no healthy tablet available for 'tablet_type:REPLICA'
Why tabletAvailableSeconds = 30s was the trigger
The operator marked Replica A as Available after 30s — well before the 9 slowest vtgates had discovered it. If tabletAvailableSeconds had been ≥ 55s (the observed worst-case discovery lag), the operator would have waited until all vtgates knew about Replica A before draining Replica B, and no errors would have occurred.
Workaround
We resolved the issue by reducing vtgate's --tablet_refresh_interval from the default 1m to 25s (below tabletAvailableSeconds = 30s):
--tablet_refresh_interval=25s
With this setting, all vtgates discover a restarted tablet within ~25s — safely before the 30s tabletAvailableSeconds window expires — and the "no healthy tablet available" errors have not recurred.
Trade-off: A shorter tablet_refresh_interval increases the polling load on the global topo server (etcd/ZooKeeper). In large clusters with many vtgate pods and keyspaces, this is measurable and may not be acceptable. The cleaner fix is to keep tablet_refresh_interval at its default and instead raise tabletAvailableSeconds so it exceeds the actual discovery time — which requires making it configurable.
Proposed Solution
Expose tabletAvailableSeconds (or a tabletAvailableTimeout duration) as an optional field on VitessClusterSpec, defaulting to 30s for backwards compatibility:
spec:
tabletAvailableTimeout: 90s # default: 30s; set to > tablet_refresh_interval
Sizing guidance: set to at least tablet_refresh_interval + a small buffer (e.g. × 1.5). With the default --tablet_refresh_interval=1m, 90s is a safe value. With a tuned --tablet_refresh_interval=25s, the current 30s default is sufficient.
References
Summary
tabletAvailableSecondsis hardcoded to30sinreconcile_tablets.go, but vtgate's default--tablet_refresh_intervalis1m. When a tablet restarts and gets a new IP, vtgates rediscover it by polling the topology — this can take up to the fulltablet_refresh_interval. Because the operator considers a tablet Available after only 30s, it proceeds to drain the next replica before most vtgates have discovered the first one, causing "no healthy tablet available fortablet_type:REPLICA" errors during rolling restarts.We would like
tabletAvailableSecondsto be exposed as a configurable field on theVitessCluster/VitessShardCRD so operators can align it with their vtgate topology poll interval without forking the operator or tuning vtgate flags.Root Cause
The timing contract that
tabletAvailableSecondsenforcestabletAvailableSecondsis meant to create a safety buffer: the operator only calls a tablet Available — and therefore safe to trigger the next rolling step — after it has been continuously Ready for at least that many seconds. The assumption is that vtgates will have finished discovering the new tablet endpoint within that window.Why the default breaks that contract
vtgate polls the topology for tablet health on the interval set by
--tablet_refresh_interval(default 1 minute). The poll is not edge-triggered. A tablet that becomes Ready just after a poll will not be discovered by vtgate until the next poll — up totablet_refresh_intervallater.With
tabletAvailableSeconds = 30sandtablet_refresh_interval = 60s(default):The exposure window =
actual discovery lag − tabletAvailableSeconds. We observed discovery lags of 45–51 seconds in production, leaving a window of up to 21 seconds where errors occur.Real Incident
Observed vtgate discovery lag
After Replica A restarted and began serving at
t=0, vtgates discovered it over a 51-second window across 10 vtgate pods:Only 1 of 10 vtgates discovered the restarted replica immediately. The remaining 9 took 44–51 seconds — consistent with
--tablet_refresh_interval=1m.How the errors occurred (two bursts)
Burst 1 (~24 errors over ~6s):
The operator considered Replica A Available ~30s after it started serving. Shortly after, the operator scheduled a drain on Replica B. When Replica B went NOT_SERVING (MySQL socket gone during pod restart), 9 of 10 vtgates had still not discovered Replica A — they saw zero REPLICA tablets.
Burst 2 (~14 errors over ~7s):
After PRS, the former primary briefly became a REPLICA and then its pod was restarted. Two vtgates that had not yet discovered Replica B as serving again had zero REPLICA tablets.
Why
tabletAvailableSeconds = 30swas the triggerThe operator marked Replica A as Available after 30s — well before the 9 slowest vtgates had discovered it. If
tabletAvailableSecondshad been ≥ 55s (the observed worst-case discovery lag), the operator would have waited until all vtgates knew about Replica A before draining Replica B, and no errors would have occurred.Workaround
We resolved the issue by reducing vtgate's
--tablet_refresh_intervalfrom the default1mto25s(belowtabletAvailableSeconds = 30s):With this setting, all vtgates discover a restarted tablet within ~25s — safely before the 30s
tabletAvailableSecondswindow expires — and the "no healthy tablet available" errors have not recurred.Trade-off: A shorter
tablet_refresh_intervalincreases the polling load on the global topo server (etcd/ZooKeeper). In large clusters with many vtgate pods and keyspaces, this is measurable and may not be acceptable. The cleaner fix is to keeptablet_refresh_intervalat its default and instead raisetabletAvailableSecondsso it exceeds the actual discovery time — which requires making it configurable.Proposed Solution
Expose
tabletAvailableSeconds(or atabletAvailableTimeoutduration) as an optional field onVitessClusterSpec, defaulting to30sfor backwards compatibility:Sizing guidance: set to at least
tablet_refresh_interval + a small buffer(e.g.× 1.5). With the default--tablet_refresh_interval=1m, 90s is a safe value. With a tuned--tablet_refresh_interval=25s, the current30sdefault is sufficient.References
reconcile_tablets.goL47–53 — hardcodedtabletAvailableSeconds = 30--tablet_refresh_interval— default1m