Worker asset upload sometimes silently fails

[numdefined]

What versions & operating system are you using?

Seen in the cf worker build pipeline:

• wrangler ^4.115.0 or ^4.119.0
• @cloudflare/workers-types: ^5.20260804.1
• pnpm/10.1.0 npm/? node/v24.18.0 linux x64

Please provide a link to a minimal reproduction

https://github.com/DonationWatch/donationwatch

Describe the Bug

Preface: it seems to only happen with lots of opennext asset cache files.

Randomly for like the last 7 days the worker build step silently fails or fails with a timeout for asset uploads:

• case of timeout being correctly propagated to the build agent:

14:45:25.448 Uploaded 9954 of 11804 assets
14:45:32.242 Uploaded 10085 of 11804 assets
15:00:19.719 Build took too long and was timed out

• case of timeout not being propagated to the build agent:

12:25:44.032 Uploaded 9585 of 11805 assets
12:25:44.355 Uploaded 9719 of 11805 assets
12:25:46.398 ▲ [WARNING] Duplicate key "options" in object literal [duplicate-object-key]
12:25:46.398
12:25:46.398     .open-next/server-functions/default/handler.mjs:259:135964:
12:25:46.398       259 │ ...nmentOffset:F3}},reset:E3}},options:[x2,y2]}),{name:"transform...
12:25:46.398           ╵                                ~~~~~~~
12:25:46.398
12:25:46.398   The original key "options" is here:
12:25:46.398
12:25:46.398     .open-next/server-functions/default/handler.mjs:259:134940:
12:25:46.398       259 │ ...ing"}),y2=[V],{name:"arrow",options:x2,async fn(a4){let{x:b2,y...
12:25:46.398           ╵                                ~~~~~~~
12:25:46.398
12:25:46.398
12:25:46.398 (node:1639) [DEP0190] DeprecationWarning: Passing args to a child process with shell option true can lead to security vulnerabilities, as the arguments are not escaped, only concatenated.
12:25:46.398 (Use `node --trace-deprecation ...` to show where the warning was created)
12:25:46.595 Success: Deploy command completed
12:25:46.771 ✨ Success! Build completed.

I would expect that the second build would also fail but it seems to just silently report as successful.

Note I'm not sure if you can reproduce it with the fake data build of the app (data:fake) but it seems to be that around ~10k assets being uploaded it times out.
When trying to reduce the amount of static routes it seems to run through just fine again:

16:29:28.424 Uploaded 5913 of 5913 assets
16:29:28.426 ✨ Success! Uploaded 5913 files (6307 already uploaded) (77.29 sec)
16:29:28.426 
16:29:29.039 Total Upload: 19260.47 KiB / gzip: 2549.69 KiB

Last time it worked before that there were around 14k assets:

18:42:00.520 Uploaded 14318 of 14410 assets
18:42:01.563 Uploaded 14410 of 14410 assets
18:42:01.580 ✨ Success! Uploaded 14410 files (6001 already uploaded) (344.69 sec)
18:42:01.580
18:42:02.092 Total Upload: 31063.32 KiB / gzip: 2786.67 KiB
18:42:07.253 Worker Startup Time: 40 ms

Please provide any relevant error logs

No response

[workers-devprod]

Note

This is an automated, advisory triage report generated by workers-devprod. It is not an official maintainer response — a maintainer will follow up.

🤖 Automated triage report

Issue Triage: #15060

Summary

Wrangler's asset upload step (as run inside the Cloudflare Workers Builds pipeline) sometimes stalls and either times out or silently stops mid-upload when a project has a very large number of assets (~11,800 OpenNext asset cache files).

Classification

• Type: Bug
• Component: packages/wrangler (Workers Assets upload path; surfaced via Workers Builds)
• Severity: High
• Has Reproduction: Yes (with link: https://github.com/DonationWatch/donationwatch)
• Quality: Complete

Findings

• Created: not present in the pre-fetched data (only updatedAt was included: 2026-08-17T21:07:07Z)
• Author: numdefined
• Version: wrangler ^4.115.0 / ^4.119.0; @cloudflare/workers-types ^5.20260804.1; pnpm 10.1.0; node v24.18.0; linux x64
• Labels: feature:workers-assets, package:deploy-helpers
• Comments: 1 (an automated advisory triage report from workers-devprod; no maintainer or reporter follow-up yet)

Analysis

• Component identification: The failure occurs during wrangler's asset upload phase (Uploaded N of 11804 assets) within the Workers Builds pipeline. Asset upload logic lives in packages/wrangler, consistent with the applied feature:workers-assets and package:deploy-helpers labels.
• Two distinct failure modes are reported: (1) timeout correctly surfaced — the build hard-stops with "Build took too long and was timed out" ~15 minutes after the last upload progress line; (2) timeout not surfaced — the upload progress simply stops (e.g. at 9,719 of 11,805) with no error, and the process appears to hang. Mode (2) is the more serious bug: a stalled upload that never produces an error or terminates cleanly.
• Reproducibility: A public reproduction repo is provided, and the reporter notes it only manifests with a high asset count (OpenNext cache files). This is a configuration-dependent but plausibly reproducible scenario for the assets team.
• Possible regression: The reporter saw this across wrangler 4.115.0 and 4.119.0 "for like the last 7 days," which hints at a possible recent regression — worth confirming against an earlier version.
• Red herring in the logs: The esbuild Duplicate key "options" warning comes from the bundled OpenNext handler (.open-next/server-functions/default/handler.mjs) and is unrelated to the upload stall; it should not distract triage.
• Prior automated triage: The single existing comment is an automated workers-devprod report that also recommended KEEP OPEN, but under a data limitation (it could not read labels/comments). That limitation does not apply here — labels and the comment were recovered — and the conclusion is unchanged.

Recommendation

Status: KEEP OPEN

Reasoning: This is a complete, well-formed bug report with a reproduction link, specific versions, and clear symptoms. The "timeout not propagated" mode is a legitimate wrangler asset-upload bug (a stalled upload that neither errors nor exits) worth engineering investigation. It should remain open for the Workers Assets team.

Action: Route to the Workers Assets / wrangler team. Attempt to reproduce with a high asset count (~11k+ files) and inspect the asset upload path for a missing or ineffective client-side timeout / retry when an individual upload request stalls. Confirm whether behavior regressed between wrangler 4.115.x and 4.119.x. The existing labels (feature:workers-assets, package:deploy-helpers) are appropriate; no label changes needed.

Suggested Labels: none (existing labels feature:workers-assets and package:deploy-helpers are already applied and appropriate)

Suggested Comment

Thanks for the detailed report and the reproduction link. To help us narrow this down, could you share a bit more?

• Does the stall reproduce when running wrangler deploy locally (outside the Workers Builds pipeline) against the same asset set, or only inside the build pipeline?
• In the case where the timeout isn't surfaced, does the process eventually exit on its own, or does it hang indefinitely until the build agent kills it?
• Is this a regression? Does an earlier wrangler version (before 4.115.0) upload the same assets successfully?

For context, the esbuild Duplicate key "options" warning in the logs comes from the bundled OpenNext handler and is unrelated to the upload stall. We'll dig into the asset upload path for the case where an upload stalls without surfacing an error or terminating.

Structured summary (JSON)

{
  "issueNumber": 15060,
  "title": "Worker asset upload sometimes silently fails",
  "githubUrl": "https://github.com/cloudflare/workers-sdk/issues/15060",
  "recommendation": "KEEP OPEN",
  "difficulty": "medium",
  "reasoning": "Complete bug report with a reproduction link and clear symptoms: wrangler's asset upload stalls and either times out or silently stops mid-upload with large asset counts (~11.8k files). The 'timeout not propagated' mode is a legitimate wrangler bug worth investigating.",
  "suggestedAction": "Route to the Workers Assets / wrangler team; reproduce with ~11k+ assets, inspect the upload path for a missing/ineffective client-side timeout on stalled requests, and check for a regression between wrangler 4.115.x and 4.119.x. Existing labels are appropriate.",
  "hasSuggestedComment": true,
  "suggestedLabels": []
}

[markws62]

This is such a critical issue. It hasn't been fixed for more than two weeks and we’re still facing the same problem. It's extremely frustrating that Cloudflare Workers remains unusable for more than 12k+ assets even after paying for the service.

[petebacondarwin]

Consolidated #14964 into this issue.

Noting the open question for whoever picks this up: it is not yet established whether the stall is client-side or service-side. The in-repo candidate is the upload queue / JWT-refresh logic in packages/deploy-helpers/src/deploy/helpers/assets.ts (~L180-320); the alternative is the asset upload service or a Builds-side timeout.

A single test should separate them — reproduce a ~12k-asset deploy with WRANGLER_LOG=debug outside the Builds pipeline:

• an upload POST visibly stalling => service side
• the queue deadlocking without surfacing an error => in-repo

I've applied package:deploy-helpers provisionally on that basis; it should be corrected if the debug run points at the service.

[markws62]

This issue is still not fixed. Stuck after 12k files uploaded. No error message, just waiting for time out.

[markws62]

Does dev team rely on AI completely to fix issues? If AI couldn't, they ignore those bugs and put in backlog.

[numdefined]

I got the false positive "successful" pipeline now also in a build with ~9k assets. Meaning the issue still persists.

12:37:52.794 Uploaded 9002 of 9345 assets
12:38:04.600 ▲ [WARNING] Duplicate key "options" in object literal [duplicate-object-key]
12:38:04.600
12:38:04.600     .open-next/server-functions/default/handler.mjs:849:128052:
12:38:04.600       849 │ ...nmentOffset:F3}},reset:E3}},options:[x2,y2]}),{name:"transform...
12:38:04.600           ╵                                ~~~~~~~
12:38:04.600
12:38:04.600   The original key "options" is here:
12:38:04.600
12:38:04.600     .open-next/server-functions/default/handler.mjs:849:127028:
12:38:04.600       849 │ ...ing"}),y2=[V],{name:"arrow",options:x2,async fn(a4){let{x:b2,y...
12:38:04.601           ╵                                ~~~~~~~
12:38:04.601
12:38:04.601
12:38:04.601 (node:1706) [DEP0190] DeprecationWarning: Passing args to a child process with shell option true can lead to security vulnerabilities, as the arguments are not escaped, only concatenated.
12:38:04.602 (Use `node --trace-deprecation ...` to show where the warning was created)
12:38:04.775 Success: Deploy command completed
12:38:05.277 ✨ Success! Build completed.

[impartshadow]

I ran a quick triage on this because it's a class of bug I track constantly: a step reports success while the work it was supposed to verify never completed — a swallowed-failure / silent-success mismatch, not a real timeout.

Your two logs pin it down: the first run properly surfaces "Build took too long and was timed out" after stalling mid-upload, but the second run stops mid-batch ("Uploaded 9719 of 11805 assets") and then jumps straight to "Success: Deploy command completed" without ever finishing the remaining ~2000 assets. That's not two different failures, it's one failure with two propagation paths. The asset uploader almost certainly issues the remaining batches concurrently and awaits them with something like Promise.all/allSettled where a rejected or hung request either throws in a place nothing catches, or gets caught and logged but not turned into a non-zero exit / thrown error before the "Success" line fires. The ~10k-asset threshold you found (works at 5,913 and 14,410 in other runs, fails intermittently in the 9-11k range) points to a request-queue/concurrency-window race rather than a hard size limit — that's consistent with a fire-and-forget branch only tripping under specific timing.

The deterministic check that would catch this before it ships: after the upload loop, assert uploadedCount === totalAssetCount (the count wrangler is already tracking and printing) and hard-fail the deploy command with a non-zero exit if they don't match, before the "Success" string is ever written. That single boundary check — output count vs. logged progress count — would have made your second log fail loudly instead of quietly.

What would confirm or kill this: a debug trace of the batch/promise settlement around asset #9700-9800 in a failing run — if you see a rejected promise near there that isn't rethrown, that's the bug.

Advisory SFA-2026-3072C4F66F · Full redacted write-up: https://impartshadow.github.io/echo-site/failure-cases/worker-asset-upload-sometimes-silently-fails-3072c4f66f.html
If this diagnosis is useful, react 👍 to this comment; if it is wrong, react 👎 — I use the verdict to choose what I investigate next.

[markws62]

It's been more than a month this issue is not resolved. Multiple users reported it. Why doesn't dev team create sample 20k files with random text to upload and debug it? It's not so difficult to create sample html files. These big projects are generally private and can't be shared publicly. Even adding sample files in addition to cloudflare docs can be easy for quick testing.