Bug
When uploading a large number of files via modal volume put, the S3 CompleteMultipartUpload POST can fail with a 503 SlowDown error. Unlike part uploads (_upload_to_s3_url) and downloads (_download_from_url), the completion POST in perform_multipart_upload has no retry logic and no 503 handling.
This means after hours of uploading (e.g. 22,677 of 22,681 files completed), the entire operation fails at the final step with no recovery.
Reproduction
modal volume put <volume-name> <local-dir> <remote-path>
With a large directory (~22K+ files), the upload completes all parts but fails on the completion step:
Error when completing multipart upload: 503
<html><head><title>503 Slow Down</title></head><body>
<h1>503 Slow Down</h1>
<ul><li>Code: SlowDown</li><li>Message: Please reduce your request rate.</li></ul>
</body></html>
Root Cause
modal/_utils/blob_utils.py - the perform_multipart_upload function makes a direct POST to the S3 completion URL without any @retry decorator or 503 handling, unlike every other S3 operation in the same file.
Expected Behavior
The completion POST should retry with exponential backoff on transient S3 errors (especially 503 SlowDown), matching the retry patterns used by _upload_to_s3_url and _download_from_url in the same file.
Bug
When uploading a large number of files via
modal volume put, the S3CompleteMultipartUploadPOST can fail with a503 SlowDownerror. Unlike part uploads (_upload_to_s3_url) and downloads (_download_from_url), the completion POST inperform_multipart_uploadhas no retry logic and no 503 handling.This means after hours of uploading (e.g. 22,677 of 22,681 files completed), the entire operation fails at the final step with no recovery.
Reproduction
With a large directory (~22K+ files), the upload completes all parts but fails on the completion step:
Root Cause
modal/_utils/blob_utils.py- theperform_multipart_uploadfunction makes a direct POST to the S3 completion URL without any@retrydecorator or 503 handling, unlike every other S3 operation in the same file.Expected Behavior
The completion POST should retry with exponential backoff on transient S3 errors (especially 503 SlowDown), matching the retry patterns used by
_upload_to_s3_urland_download_from_urlin the same file.