v0.6.19: full index still peaks at 13.4G physical footprint on 16GB macOS host

[mrm007]

Retested on cass 0.6.19 on the same macOS 16 GB host. This is a recurrence of closed #294: full indexing still drives the process to a 13 GB-class footprint.

Run context:

• cass --version: cass 0.6.19
• Command under test: cass index --full
• Host: macOS, 16 GB RAM
• Retest output dir: /Users/mrm/work/cass-0.6.16-retest/20260703T210349Z
• Monitor log: monitor.tsv

The monitor script samples top memory plus periodic vmmap -summary physical footprint. Peak observed in this run:

max top_mem:                       13G
max top_rsize:                     13G
max vmmap_physical_footprint:      13.4G
max vmmap_physical_footprint_peak: 13.4G
lowest sampled mem_free_pct:       15%

Worst sampled row:

2026-07-03T21:06:22Z	87440	02:34	2.7	19248	13G	13G	13.4G	13.4G	315392	225129192	0	0	1783112814207	1783112685295	index	15%	3615	4406	616603	20078109	23469887

Column header for that row:

ts	pid	elapsed	cpu_pct	rss_kb	top_mem	top_rsize	vmmap_physical_footprint	vmmap_physical_footprint_peak	db_bytes	wal_bytes	shm_bytes	index_bytes	lock_updated_ms	lock_last_progress_ms	lock_phase	mem_free_pct	pages_free	pages_speculative	pages_compressed	pages_swapins	pages_swapouts

Later, after lexical publish, the same run again climbed into multi-GB memory during the post-publish wedge/abort window, with samples reaching 10G-12G top memory:

2026-07-03T21:18:47Z	87440	14:58	94.3	1407568	11G	11G	11.0G	13.4G	315392	1188941392	0	340750336	1783113531425	1783113282347	index	33%	3373	719	447913	21226931	24266108
2026-07-03T21:18:53Z	87440	15:04	69.1	1287568	12G	12G	11.0G	13.4G	315392	1188941392	0	340750336	1783113534462	1783113282347	index	33%	4079	598	449821	21229964	24286716

The process eventually published the lexical index, then hit the post-publish preparing wedge and self-aborted with exit_code=70. This issue is only about the memory peak: v0.6.19 still pushes a 16 GB macOS host to the edge, and RSS alone would miss the real footprint.

Related run results:

• Lexical publish succeeded: 16096 conversations, 167766 docs.
• The post-publish wedge still reproduced; filed separately as the v0.6.10 + v0.6.11: cass index --full post-bulk-phase wedge at lexical staging→publish; cass search triggers partial recovery; only 1/8 agents ingested #282 recurrence.
• The abort-path DB/WAL integrity issue still reproduced; filed separately as the v0.6.15: killed post-publish wedge leaves tiny DB, 1GB+ WAL, stale locks, and malformed integrity state #296 recurrence.

[Dicklesworthstone]

v0.6.21 is out and is worth a memory retest: the post-publish window's multi-GB climb (your 10–12G samples during the wedge/abort phase) is expected to disappear with the frankensqlite 0.1.13 bump — the contentless-FTS INSERT path in 0.1.11 accumulated and re-tokenized the entire pending row set per insert, which is both the CPU spin in #319 and a large allocation amplifier.

The separate early peak you sampled during the main ingest phase (13.4G at ~02:34 elapsed) is not necessarily addressed by this bump; we're tracking that as its own profiling task (bead leli0) and would love a fresh monitor.tsv from a v0.6.21 run to see how much of the footprint remains once the FTS pathology is gone.

[Dicklesworthstone]

Thanks for the careful retest with the vmmap physical-footprint sampling — that's exactly the right instrument for this.

I'm treating this as a real, open memory-optimization task rather than something to hot-patch: a 13.4 GB peak on a 16 GB host during cass index --full is a genuine footprint problem (and, as you note, a recurrence of #294 — so the earlier mitigation didn't hold under the full corpus). It needs a dedicated profiling pass to find where the peak actually comes from (bulk-ingest buffering, the deferred WAL frames before the final checkpoint, in-memory index/embedding structures held across the whole run, etc.) and then bound/stream it, not a guess. That's separate from the finalize-checkpoint watchdog fix that just landed for #319/#321.

Keeping this open and tracked. If you can share the full monitor.tsv from that run it'll help pinpoint which phase drives the peak.

[mrm007]

Retested with Homebrew cass 0.6.22 on the same 16 GB macOS host.

Important caveat: v0.6.22 is e023ef9 and the #319/#321 close commit (e4360670) is one commit after that tag, so this run should not be read as a retest of the new finalize-checkpoint watchdog fix. It is still a useful memory sample for this issue.

Run details:

• run id: 20260708T-cass-0.6.22
• default data dir: /Users/mrm/Library/Application Support/com.coding-agent-search.coding-agent-search
• cass index --full exit code: 70
• lexical publish succeeded before abort: conversation_count=16152, message_count=173855, indexed_doc_count=173855
• no fts_messages warning spam observed in this run

All size units below are decimal (KB/MB/GB), not binary units.

Memory result: still peaks very high.

• max top memory: 13.96 GB
• max sampled vmmap physical footprint peak: 14.71 GB

First high peak samples, before lexical publish:

time	elapsed	cpu	rss	top mem	top rsize	vmmap footprint	vmmap peak	db	wal	index
2026-07-08T10:27:21Z	01:44	1.0%	14 MB	13.96 GB	13.96 GB	14.17 GB	14.17 GB	315 KB	235 MB	0 MB
2026-07-08T10:27:32Z	01:55	1.3%	1.93 GB	11.81 GB	11.81 GB	12.03 GB	14.71 GB	315 KB	235 MB	0 MB

Post-publish/preparing samples:

time	elapsed	cpu	rss	top mem	top rsize	vmmap footprint	vmmap peak	db	wal	index
2026-07-08T10:32:19Z	06:42	73.0%	1.91 GB	11.81 GB	11.81 GB	11.49 GB	14.71 GB	315 KB	1.209 GB	341 MB
2026-07-08T10:33:15Z	07:38	77.6%	1.75 GB	12.88 GB	12.88 GB	12.78 GB	14.71 GB	315 KB	1.217 GB	341 MB
2026-07-08T10:33:37Z	08:00	78.4%	1.71 GB	13.96 GB	13.96 GB	9.45 GB	14.71 GB	315 KB	1.220 GB	341 MB
2026-07-08T10:34:17Z	08:40	70.7%	1.29 GB	13.96 GB	13.96 GB	13.42 GB	14.71 GB	315 KB	1.222 GB	341 MB

Post-run disk/integrity state, for context:

• agent_search.db: 315 KB
• agent_search.db-wal: 1.225 GB
• index: 341 MB
• sqlite3 PRAGMA integrity_check: Error: stepping, database disk image is malformed (11)

Full run artifacts are local under cass-0.6.16-retest/20260708T-cass-0.6.22/; the monitor.tsv for this run is 13 KB if you want it attached separately.

[mrm007]

monitor.tsv

[Dicklesworthstone]

Partial mitigation shipped on main in 4fddb21 and corrected in 968a5f3: bulk import now uses a bounded WAL cap and passive checkpoints while retaining deferred-checkpoint throughput semantics. This addresses unbounded WAL growth, but it does not yet prove the reported 13.4 GB macOS peak physical-footprint problem is resolved. I am intentionally leaving this issue open until a full index run on the named 16 GB macOS workload demonstrates the actual peak RSS/physical-footprint target.