alchemy dev: Effect-hosted DO workers never register in the dev registry — cross-script DO bindings 503

[asterikx]

Symptom

Under alchemy dev (2.0.0-beta.63), a worker binding a Durable Object from another worker via scriptName gets persistent 503s:

Durable Object "EvalHost" defined in worker "eval-host-dev" not found. Make sure the worker is running locally and exports the class.

Setup: the documented container pattern — an async consumer worker + a small Effect worker hosting Cloudflare.DurableObject + Cloudflare.Container (Containers.layer), bound cross-script:

env: {
  EVAL_CONTAINER: Cloudflare.DurableObject<CfDurableObject>("EvalHost", { scriptName: `eval-host-${stage}` }),
}

Deployed stages work. Local dev worked on 2.0.0-beta.61 and is broken on 2.0.0-beta.63 (beta.62 untested here; it already contains the relevant rewrite, see below).

Live forensics (three parallel alchemy dev stacks: app, eval, opt)

1. Registry (~/.local/state/alchemy/registry/) contains entries only for the three subscriber workers (app-website-dev, eval-worker-dev, optrun-worker-dev). The two Effect host workers (eval-host-dev, opt-host-dev) have no entry at all — hence RegistryProxy.worker's resolution finds nothing and returns the 503.
2. Process table: 8 workerd processes. Only three carry --debug-port=127.0.0.1:0 — exactly the three registered workers (Runtime.start passes {"debug-port": "127.0.0.1:0"} unconditionally and its context.start(ports) performs the registry.write). The host workers exist only as their stable-port proxy instances; no real (debug-port) instance is running for them.
3. Dev output: the host workers log [host-worker] Changes detected, interrupting existing instance and the resource applies report updated — but a [host-worker] Started in …ms line never appears (the sibling API workers log [worker] Started in 3023ms normally). The hosts' runtime.start never completes.

The publish machinery itself is complete and correct in @distilled.cloud/cloudflare-runtime 0.13.5: RegistryProxyLive derives kind: "durable-object" services from worker.durableObjectNamespaces, and DurableObjectNamespace.local handles the subscribe side. The failure is upstream of it: the host worker's real instance never starts, so nothing ever publishes.

Regression window / what changed

Between beta.61 and beta.63, LocalWorkerProvider's serve path was rewritten (beta.61 src/Cloudflare/Workers/LocalWorkerProvider.ts ~L207-227 → beta.63 serveWith ~L232-280 + restartWorker + dropServeState):

• beta.61: start the new workerd first, then Effect.forkDetach(Scope.close(previous)).
• beta.63: Scope.close(previous) first (commented: "Both runtimes use the same registry key. Close the old scope first so its unregister finalizer cannot delete the replacement registration."), then Scope.fork + runtime.start, serialized per worker id via a new semaphore, with latestServes/localRuntimeState.workerRestarts powering sibling-triggered restarts (restartWorker explicitly no-ops when the worker hasn't served yet, assuming "the pending first serve will already observe the updated state").

Hypothesis (labeled as such)

The interrupt/re-serve flow around first startup loses host workers: their first serve gets interrupted (Changes detected, interrupting existing instance — e.g. the sibling Container resource updating), and nothing re-serves them afterwards — restartWorker no-ops (no latestServes entry yet, since the first serve never completed), and unlike consumer workers nothing else ever pokes them again. Net result: proxy up, real instance absent, no registry entry, every cross-script DO binding 503s. Workers that subscribe to something (queues, cross-script bindings) happen to survive, which matches the observed 3-registered/2-missing split exactly.

I couldn't pin the exact interleaving from outside — happy to run instrumented builds or provide more traces. Related: #856 (another beta.63 local-dev issue from the same setup — the capnweb RPC layer dropping Redacted in provider results; both currently block the container pattern locally, #856 patchable, this one not).

[asterikx]

Update: the regression claim in this issue is wrong — I can no longer reproduce on 2.0.0-beta.63, and the root cause turned out to be environmental on our side. Full correction below so nobody chases a phantom.

What actually happened

Our beta.63 test ran in the middle of a broken local Docker environment, which we only fully diagnosed afterwards:

1. The container images had been evicted (disk-full incident) while the provider state said "built" — host workers blocked at container bring-up on their first serve.
2. docker build was streaming a ~148 GB build context: the local dev build pipes the Dockerfile via stdin (docker build -f - <context>), where per-Dockerfile .dockerignore companions don't apply — with no root .dockerignore, the entire repo (including a huge gitignored data dir) went into every context transfer. Host-worker runtime.start sat in that transfer "forever".
3. The egress-interceptor sidecar (cloudflare/proxy-everything:3cb1195, amd64-only) crashes under qemu on Apple Silicon with Rosetta off (setsockopt: protocol not available) — killing container bring-up even once images existed. (Newer 510e5ce is multi-arch; pulling it platform-native fixes this — happy to file that separately if useful, it applies to cloudflare-tools rather than this repo.)
4. Stale alchemy … bin/exec.ts watcher processes from a crashed earlier bun dev survived and interfered with the registry.

So the observed "host workers have no --debug-port and never publish" was real, but the cause was the host's first serve never completing for environmental reasons — on beta.61 the same symptom occurred once the environment broke, which disproves the beta.61→63 serve-path-rewrite theory.

Verification on a clean environment (beta.63 + #856 applied)

• [host-worker] Started in 8629ms / Started in 7992ms — both Effect container-host workers start
• Registry contains all five workers incl. eval-host-dev.json and optrun-host-dev.json
• Cross-script DO binding resolves; full worker → DO → container round-trip answers (400 from the app payload validator, as expected for an empty probe)

What might still be worth keeping from this issue

When a host worker's first serve fails or blocks (container bring-up failure), it fails silently: no error surfaces in alchemy dev output, and nothing ever re-serves it (restartWorker no-ops without a latestServes entry). A visible error — or a retry — for a failed first serve would have cut our diagnosis from days to minutes. Feel free to retitle for that, or just close; sorry for the noise, and thanks!