Coordinate Parallel LLM Retries to Prevent Rate-Limit Retry Storms
Problem
Repo2Readme processes multiple files concurrently, while individual LLM requests already have retry and backoff behavior.
However, retries are currently coordinated at the individual request level rather than at the provider level.
When multiple workers receive rate-limit responses such as HTTP 429 at approximately the same time, each worker can independently enter its retry/backoff cycle:
Worker 1 → 429 → backoff → retry
Worker 2 → 429 → backoff → retry
Worker 3 → 429 → backoff → retry
Worker 4 → 429 → backoff → retry
Even with exponential backoff and jitter, a sufficiently parallel workload can continue generating bursts of retry requests against the same provider.
This can make rate limiting worse, increase request failures, waste retry attempts, and unnecessarily extend repository processing time.
Proposed Solution
Introduce provider-aware coordination for concurrent LLM requests and retries.
The implementation should:
- Coordinate requests going to the same LLM provider.
- Allow different providers to maintain independent limits.
- Respect provider-supplied "Retry-After" information when available.
- Prevent multiple workers from immediately retrying after a shared rate-limit event.
- Integrate with the existing retry/backoff implementation instead of creating a separate retry mechanism.
- Provide configurable concurrency/rate-limit behavior where appropriate.
- Ensure successful requests are not unnecessarily blocked by failures from unrelated providers.
- Preserve existing behavior when no provider-level rate limit is configured.
The implementation should be designed so that provider coordination does not unnecessarily serialize all LLM requests.
Testing
Add regression tests covering:
- Multiple workers receiving 429 responses do not immediately retry simultaneously.
- "Retry-After" is respected when supplied by the provider.
- Requests to different providers remain independently coordinated.
- Successful requests are not unnecessarily blocked by a rate-limited provider.
- Existing exponential backoff and jitter behavior remains functional.
- Configured concurrency/rate limits are correctly enforced under parallel workloads.
- A rate-limit event eventually allows queued requests to continue normally.
Expected Result
Parallel repository processing should remain efficient while avoiding retry storms against an LLM provider.
When a provider begins rate limiting requests, concurrent workers should coordinate their retries instead of independently generating another burst of requests.
If this aligns with the project's direction and is considered a valid issue, please assign it to me and I can work on the implementation and regression tests.
Coordinate Parallel LLM Retries to Prevent Rate-Limit Retry Storms
Problem
Repo2Readme processes multiple files concurrently, while individual LLM requests already have retry and backoff behavior.
However, retries are currently coordinated at the individual request level rather than at the provider level.
When multiple workers receive rate-limit responses such as HTTP 429 at approximately the same time, each worker can independently enter its retry/backoff cycle:
Worker 1 → 429 → backoff → retry
Worker 2 → 429 → backoff → retry
Worker 3 → 429 → backoff → retry
Worker 4 → 429 → backoff → retry
Even with exponential backoff and jitter, a sufficiently parallel workload can continue generating bursts of retry requests against the same provider.
This can make rate limiting worse, increase request failures, waste retry attempts, and unnecessarily extend repository processing time.
Proposed Solution
Introduce provider-aware coordination for concurrent LLM requests and retries.
The implementation should:
The implementation should be designed so that provider coordination does not unnecessarily serialize all LLM requests.
Testing
Add regression tests covering:
Expected Result
Parallel repository processing should remain efficient while avoiding retry storms against an LLM provider.
When a provider begins rate limiting requests, concurrent workers should coordinate their retries instead of independently generating another burst of requests.
If this aligns with the project's direction and is considered a valid issue, please assign it to me and I can work on the implementation and regression tests.