We operate a large mesh (more than 40K workloads across 10+ k8s clusters). Recently we ran into an issue that the ztunnel had trouble to get xDS pushes. The error log from istiod:
2026-03-12T04:31:40.966155Z error delta ADS: "10.216.119.31:33334" terminated with error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4200717 vs. 4194304)
istiod has a default limit of 4MB for the gRPC stream receive buffer size.
When ztunnel reconnects to istiod, ztunnel would include all its known workload names into the request. The request size is very close to the 4MB limit since we have 40K workloads.
Somehow the pod number spiked, the xDS request size exceeded the limit and the RECONNECT request failed. The xDS connection cannot self recover even if the number of pods has been back to normal level. We mitigated the issue by restarting ztunnel.
It's better to have a metric about the size of xDS request size so we can setup alerts, in order to have time to bump the limit before running into real issues.
Thanks.
We operate a large mesh (more than 40K workloads across 10+ k8s clusters). Recently we ran into an issue that the ztunnel had trouble to get xDS pushes. The error log from istiod:
istiodhas a default limit of 4MB for the gRPC stream receive buffer size.When
ztunnelreconnects toistiod,ztunnelwould include all its known workload names into the request. The request size is very close to the 4MB limit since we have 40K workloads.Somehow the pod number spiked, the xDS request size exceeded the limit and the RECONNECT request failed. The xDS connection cannot self recover even if the number of pods has been back to normal level. We mitigated the issue by restarting
ztunnel.It's better to have a metric about the size of
xDSrequest size so we can setup alerts, in order to have time to bump the limit before running into real issues.Thanks.