Repository navigation
feat: Add a retry policy to the transport - #506
Merged
Merged
Conversation
Transport::send() made exactly one HTTP attempt, so transient failures (502/503/504 from an overloaded or restarting node, refused or reset connections) reached the caller, who had to write a retry loop around every call. Retry them in the transport, on by default as in the other OpenSearch clients: up to 3 retries with exponential backoff and full jitter (100 ms base, at most 5 s of delays per request), each attempt on the next node of the connection pool. Timeouts are retried only when enabled, since the server may still apply a request that timed out. The body is serialized once and resent unchanged; request initializers and SigV4 signing run again for every attempt. Configured on TransportBuilder with max_retries, disable_retry, retry_on_status, retry_on_timeout, retry_backoff and retry_budget. tokio (time) and fastrand become regular dependencies for the backoff; both were already in the dependency graph. Closes opensearch-project#484 Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>
krocky-cooky
requested review from
VachaShah,
Xtansia and
reta
as code owners
November 1, 2026 10:47
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #506 +/- ##
==========================================
+ Coverage 56.12% 56.28% +0.15%
==========================================
Files 523 525 +2
Lines 119123 119564 +441
==========================================
+ Hits 66859 67293 +434
- Misses 52264 52271 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
reta
reviewed
Nov 3, 2026
reta
reviewed
Nov 3, 2026
reta
reviewed
Nov 3, 2026
Cap the backoff ceiling at one minute (MAX_RETRY_DELAY), so that no single delay before a retry is longer, whatever the backoff and the number of retries before it. Without a retry budget, a large backoff could otherwise wait for years; the u64::MAX fallback of the delay is no longer reachable. Hold the transport's retry policy as an Option, `Some` with the defaults unless `disable_retry` removes it, in which case a request is sent once without entering the retry loop. A retry setting called after `disable_retry` enables retries again with the defaults for the other settings. Retries stay on by default and need no configuration. Drop doc comments of the retry policy that only restated the code. Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>
Contributor
Author
|
@reta Thanks for the review! I addressed all the comments. |
reta
approved these changes
Nov 5, 2026
Collaborator
|
Thanks @krocky-cooky , I think we could bring it to |
|
The backport to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Transport::send()makes exactly one HTTP attempt, so transient failures (502/503/504 from an overloaded or restarting node, refused or reset connections) reach the caller, who has to write a retry loop around every call. This PR adds a retry policy to the transport, on by default, as in the other OpenSearch clients.What is retried
How
MultiNodeConnectionPoola retry goes to another node.Ok) or the last error is returned.Configuration
Implementation
http/retry.rs(new): the policy, i.e. defaults, what is retried, the backoff and the budget.http/transport.rs:sendprepares the body once and loops over attempts. Building and sending one attempt moved into a privatesend_once, which keeps errors building the request (not retried) apart from errors sending it.tokio(only thetimefeature added) andfastrandbecome regular dependencies, for the backoff timer and the jitter. Both were already in the dependency graph (through reqwest, and as optional dependencies ofaws-auth), so no new crates are pulled in.Testing
opensearch/tests/retry.rs(new), 14 tests against the mock server:disable_retryaws_authwith theaws-authfeature. Clippy is unchanged frommain(42 warnings).Issues Resolved
Closes #484, Part of #488
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.