[codex] Recover shard materializers on restart

[pcharbon70]

Summary

This fixes existing-cluster recovery so Bedrock restores a complete shard_materializers map after a durable restart, instead of recovering only the metadata/system materializer.

The change:

• makes RecoveryAttempt carry :shard_materializers as a first-class recovery field;
• updates MaterializerBootstrapPhase existing-cluster recovery to recover a materializer for every shard tag in the recovered shard layout;
• unlocks each recovered shard materializer with only the logs relevant to that shard and waits for catchup;
• preserves the legacy "metadata_materializer" lookup for system shard tag 0 only;
• makes log recruitment ignore non-log service tuples, including shard-tagged materializer services shaped like {:materializer, ref, shard_id};
• adds regression coverage for existing-cluster recovery restoring both system and user shard materializers.

Bug Description

Bedrock's default keyspace layout has separate shard tags for system metadata and user/data keys:

• shard tag 0 serves system keys, including \xff/system/shard_keys/*;
• shard tag 1 serves normal user/data keys.

On fresh cluster bootstrap, Bedrock already creates materializers for all shard tags in the default layout and stores them in recovery_attempt.shard_materializers. That gives topology enough information for LayoutIndex to route reads for both system and user shards.

The existing-cluster restart path was asymmetric. It recovered only the system/materializer used as metadata_materializer, queried the shard layout from it, and then advanced without rebuilding recovery_attempt.shard_materializers. TopologyPhase then copied Map.get(recovery_attempt, :shard_materializers, %{}) into the transaction system layout, producing an empty shard-materializer map after restart.

That empty map breaks non-system reads. LayoutIndex can route tag 0 through metadata_materializer, but every non-system shard tag depends on :shard_materializers. After restart, user/data shard reads can therefore have no read server even though the cluster recovered enough metadata to know the shard layout.

In practice this shows up as a durable cluster restart where metadata recovery succeeds, but normal user/data reads fail because the read-serving topology is incomplete.

Fix Details

The existing-cluster materializer bootstrap now does the recovery in two steps:

1. Recover the system shard materializer first, as before, so it can query the authoritative shard layout from system keys.
2. Extract every shard tag from that recovered layout and recover a materializer for each tag that is not already recovered.

For each shard tag, recovery now:

• finds a shard-tagged materializer service when one is available;
• creates a new materializer on a capable node when the shard materializer is missing;
• locks the materializer for the current recovery epoch;
• unlocks it with a transaction-system layout containing only logs for that shard;
• waits until it catches up to the recovery durable version;
• records the resulting pid in the shard_materializers map.

The resulting recovery attempt now enters topology with both metadata_materializer and a complete %{shard_tag => materializer_pid} map.

Compatibility Notes

The legacy "metadata_materializer" service lookup remains supported for the system shard. Non-system shards require either shard-tagged materializer service entries or materializer-capable nodes so recovery can create the missing materializers. This avoids treating the metadata materializer as a read server for unrelated user/data shards.

LogRecruitmentPhase also now filters log services by explicit log tuple shape instead of destructuring every service tuple as two elements. That keeps recovery compatible with shard-tagged materializer service tuples in available_services.

Validation

Ran locally:

• mix test test/bedrock/control_plane/director/recovery/materializer_bootstrap_phase_test.exs
• mix test test/bedrock/control_plane/director/recovery
• mix test
• mix format --check-formatted
• mix credo --strict
• git diff --check

Full suite result: 13 doctests, 158 properties, 2300 tests, 0 failures (17 excluded).

[pcharbon70]

Hey Jason, hope all is going well. I think this was an actual bug but please comment to let me know before I change the draft status.

[ty13r]

Restart-recovery findings from exercising this on a single-node metagraph store

We've been trying to use this PR to get single-node restart recovery working (metagraph's vendored bedrock 0.5.0 + this branch) and hit two layers worth sharing — the second blocks restart even with this PR applied.

1. Upstream timing race (before this PR's code runs). On restart the Coordinator becomes leader and starts the Director with an empty service_directory (director_management.ex try_to_start_director → services: t.service_directory) before the Foreman finishes its async reload→start→advertise of persisted workers. So materializer_bootstrap_phase runs with empty available_services → "System shard materializer not found, creating new one" → cluster comes up at version 0 → :bedrock_cluster_not_ready. We prototyped a bounded readiness gate in try_to_start_director (on restart: Foreman.wait_for_healthy + merge get_all_running_services into service_directory before starting the Director; bounded, fresh-start-safe). Happy to share the branch.

2. The deeper gap — the shard↔worker mapping isn't persisted. Even with workers re-advertised, find_materializer_service (materializer_bootstrap_phase.ex:~248) identifies the system materializer by shard_id == RecoveryAttempt.system_shard_id(), matching service values shaped {kind, ref, shard_id}. But on restart there's no durable record of which worker serves the system shard:

• the worker manifest is shard-less: {"id":"…","worker":"Bedrock.DataPlane.Materializer.Olivine","params":{},"cluster":"…"}
• the object-storage bootstrap doesn't record it (build_old_tsl_from_bootstrap yields empty logs/services)
• Foreman.new_worker(id, kind) takes no shard param — the shard tag is applied only at runtime in available_services (and otherwise lives inside the system materializer's own layout data — the very thing we're recovering → chicken-and-egg)

So get_all_running_services returns {id, kind, otp_name} (no shard), and recovery can't tell which reloaded materializer is tag 0. This PR's recover_existing_shard_materializers works once the system shard is found — but finding it on restart seems to need the shard identity persisted (a manifest field / a shard param on new_worker) or recovery to query each materializer for its shard.

Repro: seed a single-node tenant, then re-open the same store in a second process. Glad to help or contribute the timing-race gate if useful.

[ty13r]

Update — prototyped the gate + a stable system-materializer id; surfaces the next layer

Following up on the above: on a branch we prototyped both halves of layer 1+2 and verified against a live single-node metagraph store.

• Readiness gate in director_management.try_to_start_director (on restart: bounded Foreman.wait_for_healthy + merge get_all_running_services into service_directory before starting the Director; fresh-start unaffected, bounded/no-deadlock).
• Stable system-materializer id: create the system-shard (tag 0) materializer with the well-known id "metadata_materializer" instead of Worker.random_id(), so on restart it reloads under that key and find_materializer_service's existing legacy fallback finds it. User shards keep random ids.

Result: the "System shard materializer not found, creating new one" path is gone — recovery now reuses the persisted system materializer instead of creating an empty one.

But it surfaces the next layer: the found materializer then reports caught up to version 0 and the cluster still fails :bedrock_cluster_not_ready — i.e. the materializer's persisted state / log replay isn't being restored on restart. That part looks like core storage-server recovery semantics, which is your call to design.

Happy to share the branch (the gate + the id change, full suite green) if it's a useful starting point — and glad to defer on the data-recovery layer.