Skip to content

feat: Add a retry policy to the transport - #506

Merged
reta merged 2 commits into
opensearch-project:mainfrom
krocky-cooky:feat/transport-retry
Nov 5, 2026
Merged

reta merged 2 commits into
opensearch-project:mainfrom
krocky-cooky:feat/transport-retry

Conversation

@krocky-cooky

Copy link
Copy Markdown
Contributor

Description

Transport::send() makes exactly one HTTP attempt, so transient failures (502/503/504 from an overloaded or restarting node, refused or reset connections) reach the caller, who has to write a retry loop around every call. This PR adds a retry policy to the transport, on by default, as in the other OpenSearch clients.

What is retried

  • Responses with status 502, 503 or 504 (configurable).
  • Attempts that fail to send the request or receive the response: connection refused or reset.
  • Timeouts only when enabled, since the server may still apply a request that timed out.
  • Errors raised by request handlers (middleware) are not retried.

How

  • Up to 3 retries. The delay before retry n is drawn uniformly from zero to 100 ms × 2^(n-1) (exponential backoff with full jitter), so clients failing together don't retry together.
  • The delays of one request are bounded together (5 s by default). A retry that would go past the bound isn't made.
  • Each attempt takes the next connection of the pool, so with a MultiNodeConnectionPool a retry goes to another node.
  • The body is serialized once and sent unchanged on every attempt. Request initializers and SigV4 signing run again for each attempt, since the URL and the signing time change.
  • The body of a retried response is read before the next attempt, so its connection can be reused.
  • When the retries are exhausted, the last response (the generic client returns any status as Ok) or the last error is returned.

Configuration

let pool = MultiNodeConnectionPool::round_robin(vec![node1, node2]);
let transport = TransportBuilder::new(pool)
    .max_retries(5)                              // default 3; 0 or disable_retry() turns retries off
    .retry_on_status(&[429, 502, 503, 504])      // default 502, 503, 504
    .retry_on_timeout(true)                      // default false
    .retry_backoff(Duration::from_millis(200))   // default 100 ms; Duration::ZERO retries at once
    .retry_budget(Some(Duration::from_secs(10))) // default 5 s; None for no bound
    .build()?;

Implementation

  • http/retry.rs (new): the policy, i.e. defaults, what is retried, the backoff and the budget.
  • http/transport.rs: send prepares the body once and loops over attempts. Building and sending one attempt moved into a private send_once, which keeps errors building the request (not retried) apart from errors sending it.
  • tokio (only the time feature added) and fastrand become regular dependencies, for the backoff timer and the jitter. Both were already in the dependency graph (through reqwest, and as optional dependencies of aws-auth), so no new crates are pulled in.

Testing

  • opensearch/tests/retry.rs (new), 14 tests against the mock server:
    • retrying until success; returning the last response once retries are exhausted; 3 retries by default; disable_retry
    • statuses that aren't retried (400, 404, 429, 500); a custom status list
    • the retry resending the same method, path, query, headers and body; request initializers running for every attempt
    • failover from a node refusing connections to the next node; the error without retries
    • timeouts not retried by default, and retried when enabled
    • backoff, and a retry past the budget not being made
  • 6 unit tests of the policy (defaults, backoff ceiling and jitter, budget, handler errors).
  • The mock server can now answer with a sequence of responses and records request bodies.
  • All existing suites pass, including aws_auth with the aws-auth feature. Clippy is unchanged from main (42 warnings).

Issues Resolved

Closes #484, Part of #488

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Transport::send() made exactly one HTTP attempt, so transient failures
(502/503/504 from an overloaded or restarting node, refused or reset
connections) reached the caller, who had to write a retry loop around
every call.

Retry them in the transport, on by default as in the other OpenSearch
clients: up to 3 retries with exponential backoff and full jitter
(100 ms base, at most 5 s of delays per request), each attempt on the
next node of the connection pool. Timeouts are retried only when
enabled, since the server may still apply a request that timed out.
The body is serialized once and resent unchanged; request initializers
and SigV4 signing run again for every attempt.

Configured on TransportBuilder with max_retries, disable_retry,
retry_on_status, retry_on_timeout, retry_backoff and retry_budget.
tokio (time) and fastrand become regular dependencies for the backoff;
both were already in the dependency graph.

Closes opensearch-project#484

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>
@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.28%. Comparing base (a1deb55) to head (95d57fd).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #506      +/-   ##
==========================================
+ Coverage   56.12%   56.28%   +0.15%     
==========================================
  Files         523      525       +2     
  Lines      119123   119564     +441     
==========================================
+ Hits        66859    67293     +434     
- Misses      52264    52271       +7     
Flag Coverage Δ
integration 53.97% <ø> (-0.01%) ⬇️
unit 13.13% <ø> (+0.65%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread opensearch/src/http/retry.rs Resolved
Comment thread opensearch/src/http/transport.rs Resolved
Comment thread opensearch/src/http/transport.rs Resolved
Cap the backoff ceiling at one minute (MAX_RETRY_DELAY), so that no
single delay before a retry is longer, whatever the backoff and the
number of retries before it. Without a retry budget, a large backoff
could otherwise wait for years; the u64::MAX fallback of the delay is
no longer reachable.

Hold the transport's retry policy as an Option, `Some` with the
defaults unless `disable_retry` removes it, in which case a request is
sent once without entering the retry loop. A retry setting called after
`disable_retry` enables retries again with the defaults for the other
settings. Retries stay on by default and need no configuration.

Drop doc comments of the retry policy that only restated the code.

Signed-off-by: Takuo Kuroki <kurotaku9679.sub@gmail.com>
@krocky-cooky

Copy link
Copy Markdown
Contributor Author

@reta Thanks for the review! I addressed all the comments.

@reta
reta merged commit 601d9fb into opensearch-project:main Nov 5, 2026
39 checks passed
@reta

reta commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Thanks @krocky-cooky , I think we could bring it to 2.x as well but with None as default

@reta reta added the backport 2.x Backport to the 2.x branch label Nov 5, 2026
@opensearch-ci-bot

Copy link
Copy Markdown

The backport to 2.x failed. Please backport manually. See failed workflow run: https://github.com/opensearch-project/opensearch-rs/actions/runs/42097518612

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.x Backport to the 2.x branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add a retry policy to the transport

3 participants