The gRPC client conformance suite (task conformance:test-client-grpc-only) intermittently fails Timeouts/* test cases when the host has scheduling jitter (a shared-tenancy VM reproduces it readily; CI runners rarely do). Repeated runs of only the Timeouts cases show a ~2-3% per-case failure rate on such a host, at the same rate on main as on a feature branch, so this is a long-standing race rather than a recent regression:
FAILED: Timeouts/HTTPVersion:2/Protocol:PROTOCOL_GRPC/Codec:CODEC_PROTO/Compression:COMPRESSION_IDENTITY/TLS:false/(grpc server impl)/server-stream:
actual error {code: 13 (internal), message: "error reading response body: error reading a body from connection"} does not match expected code 4 (deadline_exceeded)
The race: the client enforces its deadline with tokio::time::timeout_at (with_deadline, connectrpc/src/client/mod.rs:1317), and the reference grpc-go server independently aborts the RPC when the same deadline expires server-side. When the server's RST_STREAM arrives before the local timer fires (timer coarseness plus scheduler delay), the in-flight body read fails first, and both body-read error sites map the failure straight to internal:
client/mod.rs:2042-2046 (failed to read response body)client/mod.rs:2662-2666 (error reading response body)
The codebase already handles the equivalent race on the trailer path: when grpc-status is missing and the deadline has elapsed, the synthesized error is upgraded to deadline_exceeded (client/mod.rs:2133-2146), with a comment citing the gRPC spec behavior of grpc-go and connect-go (RST_STREAM CANCEL is upgraded to DEADLINE_EXCEEDED after deadline elapse; connect-go checks ctx.Err() before classifying transport errors).
The fix is to apply the same upgrade at the body-read error sites: if a deadline was set and has elapsed when a transport read error surfaces, classify it as deadline_exceeded instead of internal. This would also make the conformance suite stable on jittery hosts.
The gRPC client conformance suite (
task conformance:test-client-grpc-only) intermittently failsTimeouts/*test cases when the host has scheduling jitter (a shared-tenancy VM reproduces it readily; CI runners rarely do). Repeated runs of only the Timeouts cases show a ~2-3% per-case failure rate on such a host, at the same rate onmainas on a feature branch, so this is a long-standing race rather than a recent regression:The race: the client enforces its deadline with
tokio::time::timeout_at(with_deadline,connectrpc/src/client/mod.rs:1317), and the reference grpc-go server independently aborts the RPC when the same deadline expires server-side. When the server's RST_STREAM arrives before the local timer fires (timer coarseness plus scheduler delay), the in-flight body read fails first, and both body-read error sites map the failure straight tointernal:client/mod.rs:2042-2046(failed to read response body)client/mod.rs:2662-2666(error reading response body)The codebase already handles the equivalent race on the trailer path: when
grpc-statusis missing and the deadline has elapsed, the synthesized error is upgraded todeadline_exceeded(client/mod.rs:2133-2146), with a comment citing the gRPC spec behavior of grpc-go and connect-go (RST_STREAM CANCEL is upgraded to DEADLINE_EXCEEDED after deadline elapse; connect-go checksctx.Err()before classifying transport errors).The fix is to apply the same upgrade at the body-read error sites: if a deadline was set and has elapsed when a transport read error surfaces, classify it as
deadline_exceededinstead ofinternal. This would also make the conformance suite stable on jittery hosts.