Rearm consolidation drift and bound Natural active work by gabewillen · Pull Request #6 · augmem/cortext.cpp

gabewillen · 2026-07-16T16:52:39Z

Summary

rearm consolidation hints after material cumulative throughput drawdown while keeping stable jitter quiet
keep Natural and Durable on one ingestion algorithm; Durable adds only its post-commit checkpoint/flush barrier
cut the existing SQLite HNSW route over to an F/S/T-derived production retrieval sawtooth instead of adding another route or write path
persist the consolidation activation centroid/snapshot in SQLite; behavior remains modality- and source_id-agnostic
update all algorithm/experiment records in the paper and generated manuscript

Production retrieval contract

C = lround(256 + 256F + 128S + 128T), B = lround(64 + 64F + 32S + 32T), A = 2C + 2B, R = max(2, B / 16)
successful consolidation resets effort and node budget to 8C; retrieval-active queries advance by R to 9C
activation snapshot is separately bounded by A; total query rows are bounded by 9C + A; restart opens at 9C
neutral .5/.5/.5: C=512, B=128, A=1280, R=8, floor 4096, peak 4608, total 5888; 129 is only logical B+1

Retrieval quality

per 512-query run: at most one top-1 miss (>=511/512)
nine-run aggregate top-1 >=0.999; recall@16 >=0.998; semantic coverage >=0.95
deterministic structural/rollback/provenance/work/tie invariants remain exact; misses may not cluster by modality, source, age, or knob

Result: 4,608/4,608 top-1 controls, recall@16 1.0, semantic coverage 1.0, deterministic ties, zero miss clusters, and passing text/audio/image plus shared/opaque-source invariance.

Verification

media-enabled CTest: 39,151 assertions in 688 cases, 1/1 target passed in 292.15 seconds
Python tooling: 159 tests passed
failed-recenter injection proves an aborted metadata update leaves effort/node budget unchanged
27-point structural grid (1,021 assertions); backfill/journal (1,163); restart (617); core knobs (91); source/modality invariance (2)
evidence: artifacts/flat_storage_cost/sqlite-hnsw-production-sawtooth-knob-ablation-v2.json
paper traceability: 69/69 records pass with no uncovered inventory or claim/ownership/proof-level mismatch
Quarto manuscript render and git diff --check pass

The 30,380-packet maturity run passed 511/512 top-1, recall@16 0.999265, semantic coverage 0.998571, deterministic ties, 19/20 material cycle resets, and no work-bound violations. The 15,695-packet Natural and 2,016-message Durable replays each passed 512/512 controls; the Durable horizon is insufficient for plateau proof.

Review and boundaries

recursive review repaired every round-1 and round-2 finding; fresh round-3 cumulative review is Proven at P1 with no findings or residuals
consolidation changes retrieval locality and its bounded centroid/snapshot; it does not force content-dependent write incidence or raw process time to reset
no claim of whole-engine raw-time reset, bounded whole-engine restart, Durable plateau, or production-wide boundedness
no public API/C API change; no merge, release, deployment, or publication
exact-head CI is in progress; local proof is not remote cross-platform success

Copilot

Pull request overview

This PR updates the internal consolidation-throughput hint latch so that, after a maintenance acknowledgment disarms the hint, it can be rearmed not only by recovery, but also by a material sustained throughput drawdown (with an F/S/T-derived “material range” guard) to prevent slow degradation from staying permanently silent.

Changes:

Add an F/S/T-derived drift rearm threshold (p_drift = 1 - p) and require a material relative throughput range before rearming.
Update the disarmed rearm logic to allow rearming on either recovery (z >= p_rearm) or sustained degradation (z <= p) once the excursion is material.
Extend the consolidation throughput tests to cover drift rearming and jitter suppression, and update the paper contract + regenerated manuscript.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/operations_consolidation.test.cpp	Adds monotonicity assertions for the new drift fraction and adds coverage for drift-based rearming vs. stable jitter remaining disarmed.
src/operations/consolidation_throughput_state_internal.hpp	Introduces `DriftRearmFraction()` and updates `Observe()` to gate rearming on material relative-range and allow both recovery-edge and drift-edge rearming.
docs/paper/sections/7_consolidation.qmd	Updates the consolidation contract to document `p_drift` and the new material-range + dual-edge rearm semantics.
docs/paper/sections/10_implementation.qmd	Documents the new drift-based rearm behavior and the jitter-suppression guard in the implementation section.
docs/paper/_manuscript/index.md	Regenerates the manuscript to reflect the updated consolidation contract and implementation description.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d3df85e3b1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gabewillen · 2026-07-21T05:11:29Z

Implementation/review checkpoint for exact head 6ac1e3121e4be9aa456c7ac93fdb4b09ea34023d:

Round 1 findings repaired: stale monitor/control contract, present-tense 5C paper/source claims, and missing cutover trace record.
Round 2 findings repaired: remaining 5C comments, failed-recenter ramp mutation, migration-29/31 provenance, and content-address drift.
Fresh round 3 final cumulative review: Proven at P1; no findings or residuals.
Full local proof: 39,151 assertions/688 C++ cases; 159 Python tests; 4,608/4,608 retrieval controls; 69/69 paper-trace records.
Evidence artifact SHA-256: 08e9270ce750d5247cb52bd985f7126bb570109f8246a74f12bae99617002be9.

Nonclaims remain whole-engine raw-time reset, bounded whole-engine restart, Durable plateau, production-wide boundedness, merge, release, deployment, and publication. Exact-head CI is still in progress.

gabewillen · 2026-07-21T07:27:52Z

Pushed exact-head CI/review repair as f7f0fc5.

What changed:

calibrate persisted RIF timestamps transactionally before working-memory hydration;
bound every pinned-hnswlib speculative neighbor prefetch, including empty adjacency lists, without disabling SIMD distance kernels;
make the WebAssembly row-count arithmetic explicitly std::size_t;
add deterministic RIF and empty-upper-layer HNSW regressions;
record the algorithm/proof changes in the paper and regenerate the manuscript.

Current proof:

full media CTest: 39,175 assertions in 690 cases;
focused RIF: 312 assertions in 19 cases;
HNSW CI regressions: 19 assertions in 2 cases;
Emscripten 4.0.18 full bundle: pass;
Python: 159 tests;
paper traceability: 70/70;
fresh recursive review: no P1/P2 findings on the repair tree.

The RIF review thread is replied to and resolved. Replacement exact-head CI is now the remaining gate, including Linux sanitizer proof. This does not claim whole-engine reset, bounded restart, Durable plateau, production-wide boundedness, merge, release, or deployment.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b7f9e27e01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gabewillen · 2026-07-21T08:15:08Z

Pushed follow-up native/sanitizer repair as 764e4af.

The three failed jobs shared one test issue: the regression assumed a platform-specific HNSW random-level outcome. The sanitizer job additionally found pinned hnswlib issuing a zero-byte memcpy with a potentially null empty-vector destination.

Repair:

construct the exact level-2 entry / level-1 target topology through a CORTEXT_TESTING-only seam;
preserve randomized production construction unchanged;
skip the upstream zero-byte adjacency copy;
update the paper, manuscript, and 70/70 traceability record.

Proof:

exact deterministic regression: 14 assertions;
all five HNSW regression cases: 1,184 assertions;
complete vendor patch reverse-check: pass;
fresh code review round two: no P1/P2/P3 residual;
single record review findings directly corrected as required.

Replacement exact-head CI now owns Linux native and sanitizer confirmation. No review requests or unresolved threads exist. No merge, release, deployment, bounded-restart, Durable-plateau, or production-wide boundedness claim is made.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 764e4af202

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 89a06084dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25cf7b130e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cca844d758

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2213ea42f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gabewillen · 2026-07-21T18:37:15Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f4d2e5e7c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

gabewillen · 2026-07-21T20:16:44Z

@codex review

chatgpt-codex-connector · 2026-07-21T20:25:13Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: b65b98ecea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Rearm consolidation drift and bound Natural active work

Rearm consolidation on throughput drift

29f0801

Copilot AI review requested due to automatic review settings July 16, 2026 16:52

Copilot started reviewing on behalf of gabewillen July 16, 2026 16:53 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Flatten Natural rollback snapshots

5f9b1d6

gabewillen changed the title ~~Rearm consolidation hints on throughput drift~~ Rearm consolidation drift and flatten Natural rollback Jul 16, 2026

Bound natural-retention active work

8c2385a

gabewillen changed the title ~~Rearm consolidation drift and flatten Natural rollback~~ Rearm consolidation drift and bound Natural active work Jul 18, 2026

gabewillen and others added 4 commits July 18, 2026 12:22

Align Zig SQLite feature flags

eaa366f

Repair native CI test configuration

d91adf5

Isolate emotional cascade test caches

16f6b95

Make retrieval cutover gates fail closed

d3df85e

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/signal_processor.cpp

Cut over bounded SQLite retrieval sawtooth

6ac1e31

gabewillen added 2 commits July 21, 2026 00:18

Transition retrieval cutover to PR monitoring

5f2e3b3

Repair exact-head startup and portability failures

f7f0fc5

Transition PR monitor to replacement CI

b7f9e27

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread build.zig Outdated

Comment thread src/operations/sparse_retrieval_route_sqlite_internal.cpp

Comment thread src/operations/consolidation.cpp Outdated

Make HNSW empty adjacency repair portable

764e4af

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread src/operations/sparse_retrieval_route_sqlite_internal.cpp

Comment thread src/operations/emotion_cascade.cpp

Comment thread src/operations/emotion_cascade.cpp

Comment thread src/operations/sparse_retrieval_route_sqlite_internal.cpp

Resolve PR review findings

89a0608

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread src/operations/sparse_retrieval_route_sqlite_internal.cpp Outdated

Comment thread src/operations/sparse_retrieval_route_sqlite_internal.cpp Outdated

Comment thread src/operations/consolidation.cpp

Comment thread build.zig

Resolve follow-up review findings

25cf7b1

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread src/operations/active_signal_embedding_ring_internal.hpp Outdated

Comment thread src/cortext.cpp Outdated

Comment thread src/operations/memory_storage.cpp Outdated

Fix sparse restart and hydration review findings

cca844d

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread src/signal_processor.cpp

Comment thread src/operations/rif_state_internal.hpp Outdated

Comment thread src/operations/graph_retrieval.cpp

Fix Windows checkout and review findings

d2213ea

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Fix bounded retrieval review edge cases

3f4d2e5

chatgpt-codex-connector Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread src/operations/active_signal_embedding_ring_internal.hpp

Comment thread src/operations/memory_storage.cpp

Comment thread src/operations/sparse_retrieval_route_sqlite_internal.cpp

Fix exact sparse recovery edges

b65b98e

gabewillen merged commit 2353640 into main Jul 21, 2026
12 checks passed

gabewillen deleted the fix/consolidation-drift-rearm branch July 21, 2026 20:33

gabewillen added a commit that referenced this pull request Jul 24, 2026

Merge pull request #6 from augmem/fix/consolidation-drift-rearm

fabe15b

Rearm consolidation drift and bound Natural active work

Uh oh!

Conversation

gabewillen commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Production retrieval contract

Retrieval quality

Verification

Review and boundaries

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

gabewillen commented Jul 21, 2026

Uh oh!

gabewillen commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabewillen commented Jul 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabewillen commented Jul 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

gabewillen commented Jul 16, 2026 •

edited

Loading

gabewillen commented Jul 21, 2026 •

edited

Loading