Summary
After enrolling a namespace into ambient mode, the average MySQL connection-establishment time to external (non-mesh) database servers increased by ~0.5 ms per connection. Steady-state latency on established connections (query execution) was unchanged over the same period.
For a client that opens a new connection per request rather than pooling, this is a per-request cost. I would like to understand whether ~0.5 ms is the expected cost of the passthrough path, and whether there is any way to tune or narrow it.
Environment
- Istio 1.30.2, ambient mode (istiod + ztunnel DaemonSet + istio-cni, in-pod redirection —
base, istiod, cni and ztunnel charts all on 1.30.2)
- Cilium 1.19.1, with
socketLB.enabled: true and socketLB.hostNamespaceOnly: true (socket-LB restricted to the host namespace so ClusterIPs survive into the datapath, which ambient requires for service-attached waypoints)
- Ambient DNS capture disabled
- Destinations are external, non-mesh MySQL servers outside the cluster. No
ServiceEntry, no waypoint on this path, so ztunnel handles them as passthrough — telemetry reports connection_security_policy="unknown" with all destination labels unknown.
- Client opens its own connections per request; no connection pooling.
Cilium predates the ambient enrollment and was unchanged across the before/after comparison, so it is not a candidate cause — included only because socket-LB coverage affects the ambient datapath.
Observations
- The increase is ~0.5 ms per connection, and it is consistent across several different external database destinations — additive rather than proportional to the baseline connect time.
- Query latency on established connections did not change. The regression is specific to connection establishment, not to throughput or per-byte cost.
What we ruled out
- Not ztunnel CPU contention. Zero CFS throttling on any ztunnel pod on any cluster; CPU and memory both comfortably below their limits.
- Not queueing. A synthetic probe shows a very tight latency distribution with no tail, which is what I would expect from a structural cost rather than contention.
ZTUNNEL_WORKER_THREADS is unset (default). Happy to test a specific value if that is worth ruling out.
Working hypothesis
The cost may be per round trip rather than per connection, and a MySQL login is unusually chatty:
| Phase |
Round trips |
| TCP handshake |
1 |
| Server greeting (server speaks first, unprompted) |
~0.5 |
| Client auth response → server OK |
1 |
| TLS upgrade, when the client negotiates it |
+2 |
Driver session init (SET NAMES, charset/sql_mode probes) |
+1–2 |
So 3–4 round trips minimum, 6–8 with TLS — each one crossing ztunnel in both directions. At roughly 0.15–0.2 ms per proxy traversal, 3–4 round trips reproduces the observed ~0.5 ms. If that is right, the number is expected behaviour for a userspace proxy hop, and the cost any given protocol pays is set by how chatty its handshake is rather than by anything that can be optimized away in ztunnel. A protocol that authenticates in a single round trip would barely notice; MySQL, Postgres and similar pay several times over.
Questions
- Is ~0.5 ms the expected added cost for ztunnel passthrough of a multi-round-trip connection handshake to an external destination, or is it higher than it should be?
- Is there any tuning that reduces per-traversal latency on the passthrough path (worker threads, splice behaviour, socket options)?
- Is per-port or per-destination-CIDR exclusion from capture on the roadmap? Per Ambient Mode ignores tcpKeepalive for External TCP Services and fails to bypass ports via annotations (AWS EKS Dual-Stack) istio#58954 (the same problem reported for PostgreSQL/5432) the sidecar bypass annotations are not valid in ambient, so the only options today are pod-level
istio.io/dataplane-mode=none or CNI excludeNamespaces — both all-or-nothing. Where most of a workload's outbound connections go to non-mesh destinations (and so receive no mTLS, identity or authorization from the mesh), a narrower exclusion would allow keeping the mesh for the traffic that actually benefits from it.
Possibly useful for others measuring this
In an enrolled pod, connect() always succeeds — connecting to a closed port on an unreachable host returns success in ~0.14 ms, because ztunnel accepts locally before dialling upstream. A by-IP connect() measures ~40 µs, i.e. just the loopback handshake with ztunnel. Connect-time instrumentation therefore cannot observe this cost at all; it surfaces later, on first read.
Anyone measuring reachability or latency in ambient should compare time to first byte against an equivalent pod labelled istio.io/dataplane-mode=none on the same node, rather than connect time. For MySQL specifically this is easy and needs no credentials, since the server sends its greeting unprompted: connect, read the first bytes, close. (Do that against a serving pod rather than an idle one — bare connects that never complete a login count toward MySQL's per-host error counter, and at max_connect_errors the server blocks the client IP.)
This tripped us up for a while and may be worth a note in the ambient troubleshooting docs.
Summary
After enrolling a namespace into ambient mode, the average MySQL connection-establishment time to external (non-mesh) database servers increased by ~0.5 ms per connection. Steady-state latency on established connections (query execution) was unchanged over the same period.
For a client that opens a new connection per request rather than pooling, this is a per-request cost. I would like to understand whether ~0.5 ms is the expected cost of the passthrough path, and whether there is any way to tune or narrow it.
Environment
base,istiod,cniandztunnelcharts all on 1.30.2)socketLB.enabled: trueandsocketLB.hostNamespaceOnly: true(socket-LB restricted to the host namespace so ClusterIPs survive into the datapath, which ambient requires for service-attached waypoints)ServiceEntry, no waypoint on this path, so ztunnel handles them as passthrough — telemetry reportsconnection_security_policy="unknown"with all destination labelsunknown.Cilium predates the ambient enrollment and was unchanged across the before/after comparison, so it is not a candidate cause — included only because socket-LB coverage affects the ambient datapath.
Observations
What we ruled out
ZTUNNEL_WORKER_THREADSis unset (default). Happy to test a specific value if that is worth ruling out.Working hypothesis
The cost may be per round trip rather than per connection, and a MySQL login is unusually chatty:
SET NAMES, charset/sql_mode probes)So 3–4 round trips minimum, 6–8 with TLS — each one crossing ztunnel in both directions. At roughly 0.15–0.2 ms per proxy traversal, 3–4 round trips reproduces the observed ~0.5 ms. If that is right, the number is expected behaviour for a userspace proxy hop, and the cost any given protocol pays is set by how chatty its handshake is rather than by anything that can be optimized away in ztunnel. A protocol that authenticates in a single round trip would barely notice; MySQL, Postgres and similar pay several times over.
Questions
istio.io/dataplane-mode=noneor CNIexcludeNamespaces— both all-or-nothing. Where most of a workload's outbound connections go to non-mesh destinations (and so receive no mTLS, identity or authorization from the mesh), a narrower exclusion would allow keeping the mesh for the traffic that actually benefits from it.Possibly useful for others measuring this
In an enrolled pod,
connect()always succeeds — connecting to a closed port on an unreachable host returns success in ~0.14 ms, because ztunnel accepts locally before dialling upstream. A by-IPconnect()measures ~40 µs, i.e. just the loopback handshake with ztunnel. Connect-time instrumentation therefore cannot observe this cost at all; it surfaces later, on first read.Anyone measuring reachability or latency in ambient should compare time to first byte against an equivalent pod labelled
istio.io/dataplane-mode=noneon the same node, rather than connect time. For MySQL specifically this is easy and needs no credentials, since the server sends its greeting unprompted: connect, read the first bytes, close. (Do that against a serving pod rather than an idle one — bare connects that never complete a login count toward MySQL's per-host error counter, and atmax_connect_errorsthe server blocks the client IP.)This tripped us up for a while and may be worth a note in the ambient troubleshooting docs.