Skip to content

Commit f85f140

Browse files
flyingrobotsflyingrobots
authored
Merge pull request #256 from flyingrobots/materialization-bus
feat(materialization): complete Phase 3 with FinalizeReport and cross-platform DIND
2 parents 7430a8e + 549cfda commit f85f140

29 files changed

Lines changed: 6316 additions & 142 deletions

‎.github/workflows/determinism.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
# © James Ross Ω FLYING•ROBOTS <https://github.com/flyingrobots>
3+
name: Determinism Tests
4+
5+
on:
6+
push:
7+
branches:
8+
- main
9+
pull_request:
10+
11+
permissions:
12+
contents: read
13+
14+
jobs:
15+
determinism:
16+
name: Materialization Determinism
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v4
20+
with:
21+
submodules: false
22+
- uses: dtolnay/rust-toolchain@1.90.0
23+
- uses: Swatinem/rust-cache@v2
24+
with:
25+
workspaces: |
26+
.
27+
- name: Run determinism tests
28+
run: cargo test --package warp-core --test materialization_determinism

‎.github/workflows/dind-cross-platform.yml‎

Lines changed: 273 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
# © James Ross Ω FLYING•ROBOTS <https://github.com/flyingrobots>
3+
name: DIND Cross-Platform Determinism
4+
5+
on:
6+
schedule:
7+
# Weekly: Sunday at 6am UTC (macOS runners are expensive)
8+
- cron: "0 6 * * 0"
9+
workflow_dispatch: # Allow manual triggering
10+
11+
jobs:
12+
build-and-hash:
13+
name: DIND (${{ matrix.platform }})
14+
strategy:
15+
fail-fast: false
16+
matrix:
17+
include:
18+
- os: ubuntu-24.04
19+
platform: linux-x64
20+
- os: ubuntu-24.04-arm
21+
platform: linux-arm64
22+
- os: windows-2022
23+
platform: windows-x64
24+
- os: macos-15
25+
platform: macos-arm64
26+
runs-on: ${{ matrix.os }}
27+
steps:
28+
- uses: actions/checkout@v4
29+
with:
30+
submodules: false
31+
32+
- uses: dtolnay/rust-toolchain@1.90.0
33+
34+
- uses: Swatinem/rust-cache@v2
35+
with:
36+
workspaces: |
37+
.
38+
39+
- name: Build DIND harness
40+
run: cargo build -p echo-dind-harness --release
41+
42+
- name: Run DIND PR suite and collect hashes
43+
shell: bash
44+
run: |
45+
set -euo pipefail
46+
47+
mkdir -p artifacts
48+
49+
# Find all .eintlog files with 'pr' tag in MANIFEST
50+
MANIFEST="testdata/dind/MANIFEST.json"
51+
if [[ ! -f "$MANIFEST" ]]; then
52+
echo "ERROR: MANIFEST.json not found at $MANIFEST"
53+
exit 1
54+
fi
55+
56+
# Use node to extract PR scenarios (cross-platform JSON parsing)
57+
node -e "
58+
const fs = require('fs');
59+
const manifest = JSON.parse(fs.readFileSync('$MANIFEST', 'utf8'));
60+
const prScenarios = manifest.filter(s => s.tags && s.tags.includes('pr'));
61+
prScenarios.forEach(s => console.log(s.path));
62+
" > /tmp/pr-scenarios.txt
63+
64+
echo "=== PR Scenarios ==="
65+
cat /tmp/pr-scenarios.txt
66+
echo "===================="
67+
68+
while IFS= read -r scenario_file; do
69+
[[ -z "$scenario_file" ]] && continue
70+
71+
scenario_path="testdata/dind/$scenario_file"
72+
if [[ ! -f "$scenario_path" ]]; then
73+
echo "WARNING: Scenario file not found: $scenario_path"
74+
continue
75+
fi
76+
77+
# Derive output filename: foo.eintlog -> foo.hashes.json
78+
base_name="${scenario_file%.eintlog}"
79+
out_file="artifacts/${base_name}.hashes.json"
80+
81+
echo ">>> Recording: $scenario_path -> $out_file"
82+
cargo run -p echo-dind-harness --release --quiet -- record "$scenario_path" --out "$out_file"
83+
84+
done < /tmp/pr-scenarios.txt
85+
86+
echo "=== Generated hash files ==="
87+
ls -la artifacts/
88+
echo "============================"
89+
90+
- name: Upload hash artifacts
91+
uses: actions/upload-artifact@v4
92+
with:
93+
name: dind-hashes-${{ matrix.platform }}
94+
path: artifacts/*.hashes.json
95+
if-no-files-found: error
96+
retention-days: 7
97+
98+
verify-identical:
99+
name: Verify Cross-Platform Determinism
100+
needs: build-and-hash
101+
runs-on: ubuntu-latest
102+
steps:
103+
- uses: actions/checkout@v4
104+
with:
105+
submodules: false
106+
107+
- name: Download all artifacts
108+
uses: actions/download-artifact@v4
109+
with:
110+
path: all-artifacts
111+
112+
- name: List downloaded artifacts
113+
run: |
114+
echo "=== Downloaded artifacts ==="
115+
find all-artifacts -type f -name "*.json" | sort
116+
echo "============================="
117+
118+
- name: Semantic comparison of hash files
119+
shell: bash
120+
run: |
121+
set -euo pipefail
122+
123+
# Create the comparison script
124+
node << 'COMPARE_SCRIPT'
125+
const fs = require('fs');
126+
const path = require('path');
127+
128+
// Discover all platform directories
129+
const artifactsDir = 'all-artifacts';
130+
const platformDirs = fs.readdirSync(artifactsDir)
131+
.filter(d => d.startsWith('dind-hashes-'))
132+
.map(d => ({
133+
name: d.replace('dind-hashes-', ''),
134+
path: path.join(artifactsDir, d)
135+
}));
136+
137+
if (platformDirs.length < 2) {
138+
console.error('ERROR: Need at least 2 platforms to compare, found:', platformDirs.length);
139+
process.exit(1);
140+
}
141+
142+
console.log(`Found ${platformDirs.length} platforms:`, platformDirs.map(p => p.name).join(', '));
143+
144+
// Build a map of scenario -> platform -> data
145+
const scenarioMap = new Map();
146+
147+
for (const platform of platformDirs) {
148+
const files = fs.readdirSync(platform.path).filter(f => f.endsWith('.hashes.json'));
149+
for (const file of files) {
150+
const filePath = path.join(platform.path, file);
151+
const data = JSON.parse(fs.readFileSync(filePath, 'utf8'));
152+
153+
if (!scenarioMap.has(file)) {
154+
scenarioMap.set(file, new Map());
155+
}
156+
scenarioMap.get(file).set(platform.name, { path: filePath, data });
157+
}
158+
}
159+
160+
console.log(`\nFound ${scenarioMap.size} scenarios to compare\n`);
161+
162+
let failures = [];
163+
let successes = 0;
164+
165+
for (const [scenario, platforms] of scenarioMap) {
166+
console.log(`=== Comparing: ${scenario} ===`);
167+
168+
const platformNames = Array.from(platforms.keys());
169+
if (platformNames.length < 2) {
170+
console.log(` WARNING: Only found on ${platformNames.length} platform(s), skipping`);
171+
continue;
172+
}
173+
174+
const baseline = platforms.get(platformNames[0]);
175+
const baselineData = baseline.data;
176+
let scenarioFailed = false;
177+
178+
for (let i = 1; i < platformNames.length; i++) {
179+
const other = platforms.get(platformNames[i]);
180+
const otherData = other.data;
181+
const errors = [];
182+
183+
// Compare metadata fields
184+
if (baselineData.elog_version !== otherData.elog_version) {
185+
errors.push(`elog_version: ${baselineData.elog_version} vs ${otherData.elog_version}`);
186+
}
187+
if (baselineData.schema_hash_hex !== otherData.schema_hash_hex) {
188+
errors.push(`schema_hash_hex: ${baselineData.schema_hash_hex} vs ${otherData.schema_hash_hex}`);
189+
}
190+
if (baselineData.hash_domain !== otherData.hash_domain) {
191+
errors.push(`hash_domain: ${baselineData.hash_domain} vs ${otherData.hash_domain}`);
192+
}
193+
if (baselineData.hash_alg !== otherData.hash_alg) {
194+
errors.push(`hash_alg: ${baselineData.hash_alg} vs ${otherData.hash_alg}`);
195+
}
196+
197+
// Compare hashes array length
198+
let firstDivergenceStep = null;
199+
if (baselineData.hashes_hex.length !== otherData.hashes_hex.length) {
200+
errors.push(`hashes_hex.length: ${baselineData.hashes_hex.length} vs ${otherData.hashes_hex.length}`);
201+
} else {
202+
// Compare each hash - find first divergence
203+
for (let j = 0; j < baselineData.hashes_hex.length; j++) {
204+
if (baselineData.hashes_hex[j] !== otherData.hashes_hex[j]) {
205+
if (firstDivergenceStep === null) {
206+
firstDivergenceStep = {
207+
step: j,
208+
baseline: baselineData.hashes_hex[j],
209+
other: otherData.hashes_hex[j]
210+
};
211+
}
212+
errors.push(`hashes_hex[${j}]: ${baselineData.hashes_hex[j]} vs ${otherData.hashes_hex[j]}`);
213+
// Only report first few divergences per comparison
214+
if (errors.length > 5) {
215+
errors.push(`... (truncated, more divergences exist)`);
216+
break;
217+
}
218+
}
219+
}
220+
}
221+
222+
if (errors.length > 0) {
223+
scenarioFailed = true;
224+
failures.push({
225+
scenario,
226+
baseline: platformNames[0],
227+
other: platformNames[i],
228+
errors,
229+
firstDivergenceStep
230+
});
231+
console.log(` DIVERGENCE: ${platformNames[0]} vs ${platformNames[i]}`);
232+
if (firstDivergenceStep !== null) {
233+
console.log(` >>> FIRST DIVERGENCE AT STEP ${firstDivergenceStep.step}`);
234+
console.log(` ${platformNames[0]}: ${firstDivergenceStep.baseline}`);
235+
console.log(` ${platformNames[i]}: ${firstDivergenceStep.other}`);
236+
}
237+
errors.forEach(e => console.log(` - ${e}`));
238+
} else {
239+
console.log(` OK: ${platformNames[0]} == ${platformNames[i]}`);
240+
}
241+
}
242+
243+
if (!scenarioFailed) {
244+
successes++;
245+
console.log(` PASS: All ${platformNames.length} platforms identical`);
246+
}
247+
console.log('');
248+
}
249+
250+
console.log('='.repeat(60));
251+
console.log(`SUMMARY: ${successes} scenarios passed, ${failures.length} comparisons failed`);
252+
console.log('='.repeat(60));
253+
254+
if (failures.length > 0) {
255+
console.error('\n!!! CROSS-PLATFORM DETERMINISM FAILURE !!!\n');
256+
for (const f of failures) {
257+
console.error(`Scenario: ${f.scenario}`);
258+
console.error(` Baseline: ${f.baseline}`);
259+
console.error(` Divergent: ${f.other}`);
260+
if (f.firstDivergenceStep !== null) {
261+
console.error(` >>> FIRST DIVERGENCE AT STEP ${f.firstDivergenceStep.step}`);
262+
console.error(` ${f.baseline}: ${f.firstDivergenceStep.baseline}`);
263+
console.error(` ${f.other}: ${f.firstDivergenceStep.other}`);
264+
}
265+
console.error(` All errors:`);
266+
f.errors.forEach(e => console.error(` - ${e}`));
267+
console.error('');
268+
}
269+
process.exit(1);
270+
}
271+
272+
console.log('\nAll platforms produced identical hashes. Determinism verified!');
273+
COMPARE_SCRIPT

‎AGENTS.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,36 +5,42 @@
55
Welcome to the **Echo** project. This file captures expectations for any LLM agent (and future-human collaborator) who touches the repo.
66

77
## Core Principles
8+
89
- **Honor the Vision**: Echo is a deterministic, multiverse-aware ECS. Consult `docs/architecture-outline.md` before touching runtime code.
910
- **Document Ruthlessly**: Every meaningful design choice should land in `docs/` (specs, diagrams, ADRs) or PR descriptions.
1011
- **Docstrings Aren't Optional**: Public APIs across crates (`warp-core`, `warp-ffi`, `warp-wasm`, etc.) must carry rustdoc comments that explain intent, invariants, and usage. Treat missing docs as a failing test.
1112
- **Determinism First**: Avoid introducing sources of nondeterminism without a mitigation plan.
1213
- **Temporal Mindset**: Think in timelines—branching, merging, entropy budgets. Feature work should map to Chronos/Kairos/Aion axes where appropriate.
1314

1415
## Timeline Logging
16+
1517
- Capture milestones, blockers, and decisions in relevant specs, ADRs, or PR descriptions.
1618
- AGENTS.md and `TASKS-DAG.md` are append-only; see `docs/append-only-invariants.md` plus `scripts/check-append-only.js` for the enforcement plan that CI will run before merges.
1719

1820
## Workflows & Automation
21+
1922
- The contributor playbook lives in `docs/workflows.md` (policy + blessed commands + automation).
2023
- Preferred repo maintenance entrypoint is `cargo xtask …` (see `xtask/` and `.cargo/config.toml`).
2124
- Planning DAG artifacts live in `docs/assets/dags/` and are documented in `docs/dependency-dags.md`.
2225
- For automated DAG refresh PRs, set `DAG_REFRESH_ISSUE=<issue-number>` as a GitHub Actions variable so the bot PR body includes `Refs #…`.
2326

2427
## Repository Layout
25-
- `packages/echo-core`: Runtime core (ECS, scheduler, Codex’s Baby, timelines).
28+
29+
- `crates/warp-core`: Runtime core (WARP graph model, materialization bus).
2630
- `apps/playground`: Vite sandbox and inspector (future).
2731
- `docs/`: Specs, diagrams, memorials.
2832
- `docs/notes`: Working notes and explorations (non-authoritative).
2933

3034
## Working Agreement
35+
3136
- **Isolated Branches**: Every new task, feature, or bugfix **MUST** begin on a fresh, isolated branch based on the latest `main` (unless context explicitly dictates otherwise). Never mix unrelated objectives on the same branch.
3237
- Keep `main` pristine. Feature work belongs on branches named `echo/<feature>` or `timeline/<experiment>`.
3338
- Tests and benchmarks are mandatory for runtime changes once the harness exists.
3439
- Respect determinism: preferably no random seeds without going through the Echo PRNG.
3540
- Run `cargo clippy --all-targets -- -D missing_docs` and `cargo test` before every PR; CI will expect a zero-warning, fully documented surface.
3641

3742
### PRs & Issues (Linkage Policy)
43+
3844
- Every PR must be tied to a GitHub Issue.
3945
- If no suitable issue exists, open one before you open the PR.
4046
- Use explicit closing keywords in the PR body: include a line like `Closes #<issue-number>` so the issue auto‑closes on merge.
@@ -43,6 +49,7 @@ Welcome to the **Echo** project. This file captures expectations for any LLM age
4349
- Project hygiene: assign the PR's linked issue to the correct Milestone and Board column (Blocked/Ready/Done) as part of the PR.
4450

4551
### Git Hooks & Local CI
52+
4653
- Install repo hooks once with `make hooks` (configures `core.hooksPath`).
4754
- Formatting: pre-commit auto-fixes with `cargo fmt` by default. Set `ECHO_AUTO_FMT=0` to run check-only instead.
4855
- Toolchain: pre-commit verifies your active toolchain matches `rust-toolchain.toml`.
@@ -52,6 +59,7 @@ Welcome to the **Echo** project. This file captures expectations for any LLM age
5259
Use the repository scripts/hooks; do not add dual-license headers to code.
5360

5461
## Git Real
62+
5563
1. **NEVER** use `--force` with any git command. If you think you need it, stop and ask the human for help.
5664
2. **NEVER** use rebase. Embrace messy distributed history; plain merges capture the truth, rebases rewrite it.
5765
3. **NEVER** amend a commit. Make a new commit instead of erasing recorded history.

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,23 @@
44

55
## Unreleased
66

7+
## 2026-01-17 — MaterializationBus Phase 3 Complete
8+
9+
- Completed MaterializationBus Phase 3 implementation:
10+
- FinalizeReport pattern: `finalize()` never fails, returns `{channels, errors}`
11+
- Prevents silent data loss when one channel has StrictSingle conflict
12+
- 7 new SPEC Police tests for conflict preservation
13+
- Added new modules to `warp-core/src/materialization`:
14+
- `emission_port.rs` — Port abstraction for emission routing
15+
- `reduce_op.rs` — Reduction operation definitions
16+
- `scoped_emitter.rs` — Scoped emission context management
17+
- Added CI workflows:
18+
- `determinism.yml` — PR-gated determinism tests
19+
- `dind-cross-platform.yml` — Weekly cross-platform determinism proof (Linux x64/ARM64, Windows, macOS)
20+
- Added tooling:
21+
- `cargo xtask dind` command with `run`, `record`, `torture`, and `converge` subcommands
22+
- DIND mission 100% complete.
23+
724
- Added `codec` module to `echo-wasm-abi`:
825
- Deterministic binary codec (`Reader`/`Writer`) for length-prefixed LE scalars
926
- Q32.32 fixed-point helpers (`fx_from_i64`, `fx_from_f32`, `vec3_fx_from_*`)

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)