Faster upload token refresh on 5xx errors during uploads #548
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a server returns a 5xx error (e.g., 503 Service Unavailable) during file upload, the SDK now properly refreshes upload tokens instead of retrying with stale cached upload URLs a bunch of times first. Previously, the HTTP layer would catch ServiceError and retry using the same pre-fetched upload URLs from the token pool. This caused repeated failures because the upload URLs may no longer be valid after a 5xx error and it delayed the recovery. The fix introduces ServiceErrorDuringUpload exception that disables HTTP-level retries (similar to B2RequestTimeoutDuringUpload), forcing the error to propagate to the upload manager. The upload manager then clears cached tokens via clear_bucket_upload_data() and requests fresh upload URLs for retry attempts. Changes: - Added ServiceErrorDuringUpload exception class - Convert ServiceError to ServiceErrorDuringUpload in b2http.py - Fixed RawSimulator to generate unique upload tokens per URL - Added comprehensive test for token refresh on 503 errors Fixes Backblaze/B2_Command_Line_Tool#1118
| # Convert ServiceError to ServiceErrorDuringUpload for upload operations. | ||
| # This disables HTTP-level retries and forces the upload manager to clear | ||
| # cached upload tokens and request fresh ones. | ||
| raise ServiceErrorDuringUpload(str(e)) |
There was a problem hiding this comment.
The method post_content_return_json is being called by _post_json_return_json, and the latter is being used by almost all post requests.
I am afraid this will break retries for all POST requests, not just ones related to uploads.
There was a problem hiding this comment.
Yeah that would be terrible. Please make sure we have tests for this (I assumed that we do and that the fact that the tests pass after the fix means it's ok, but maybe not?)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a server returns a 5xx error (e.g., 503 Service Unavailable) during file upload, the SDK now properly refreshes upload tokens instead of retrying with stale cached upload URLs a bunch of times first.
Previously, the HTTP layer would catch ServiceError and retry using the same pre-fetched upload URLs from the token pool. This caused repeated failures because the upload URLs may no longer be valid after a 5xx error and it delayed the recovery.
The fix introduces ServiceErrorDuringUpload exception that disables HTTP-level retries (similar to B2RequestTimeoutDuringUpload), forcing the error to propagate to the upload manager. The upload manager then clears cached tokens via clear_bucket_upload_data() and requests fresh upload URLs for retry attempts.
Changes:
Fixes Backblaze/B2_Command_Line_Tool#1118