status reports storage_integrity=ok after stock SQLite quick_check proves committed autoindex corruption

[etafund]

Summary

On current CASS main, cass status --json can report:

{
  "healthy": true,
  "status": "healthy",
  "storage_integrity": {
    "storage_state": "ok",
    "source_of_truth_risk": "none",
    "archive_readability": "readable",
    "checks_attempted": [
      {
        "name": "db_open",
        "timed_out": false,
        "read_only": true
      }
    ]
  }
}

while stock SQLite's PRAGMA quick_check on both the live database and a consistent .backup snapshot reports committed structural corruption:

Error: stepping, database disk image is malformed (11)
*** in database main ***
Page 41: never used
wrong # of entries in index sqlite_autoindex_conversations_1

The immediate bug is not that the lightweight status command omits a multi-gigabyte integrity scan. The bug is that it converts only db_open succeeded into the definitive verdicts storage_state: "ok" and source_of_truth_risk: "none".

For operators and coding agents, ok means a check passed. Here, the relevant check was never attempted.

Environment

CASS commit:        968a5f3c914e975327c998368ab2c5510aa22654
Build:              cargo build --release --locked
Platform:           Linux x86_64
Database size:      approximately 6.2 GiB
Canonical rows:     9,596 conversations / 1,209,091 messages
Storage dependency: fsqlite 0.1.13

All paths and corpus content are intentionally omitted. The database is a large multi-connector agent-session archive.

Reproduction

1. Use a CASS database whose canonical table remains readable but whose unique autoindex is malformed. In the observed file:
   • a table scan of conversations succeeds;
   • forcing sqlite_autoindex_conversations_1 returns SQLITE_CORRUPT;
   • stock SQLite quick_check reports one orphan page and a wrong autoindex entry count.
2. Run:

cass status --json > status.json
sqlite3 -readonly "$DATA_DIR/agent_search.db" 'PRAGMA quick_check;'

3. Compare:

jq '.healthy, .status, .storage_integrity' status.json

Observed CASS result:

{
  "storage_state": "ok",
  "source_of_truth_risk": "none",
  "archive_readability": "readable",
  "checks_attempted": [
    {
      "name": "db_open",
      "elapsed_ms": 0,
      "timed_out": false,
      "read_only": true
    }
  ]
}

Observed stock SQLite result:

database disk image is malformed (11)
Page 41: never used
wrong # of entries in index sqlite_autoindex_conversations_1

The same quick_check result occurs on a snapshot created with stock SQLite .backup, ruling out a transient read of an incompletely copied file. The live table remains scan-readable, which explains why db_open and simple counts can succeed.

Why this matters

This is a truth-surface contract problem:

• an agent sees healthy: true, storage_state: ok, and source_of_truth_risk: none;
• it reasonably skips recovery and may continue writing to a malformed canonical store;
• the actual evidence only supports "database opens and bounded reads work";
• search can remain complete because the authoritative lexical index was already published, masking canonical storage damage.

A false definitive success is worse than an explicit "unchecked" state.

Expected behavior

When status only performs db_open, report the scope honestly. For example:

{
  "storage_integrity": {
    "storage_state": "unchecked",
    "source_of_truth_risk": "unknown",
    "archive_readability": "readable",
    "checks_attempted": [
      {"name": "db_open", "result": "pass"}
    ],
    "checks_not_attempted": [
      {"name": "quick_check", "reason": "outside_status_budget"}
    ]
  }
}

If a recent, fingerprint-matched integrity attestation exists, status could safely project that cached verdict with its age and database fingerprint. It should not synthesize ok from openability alone.

Suggested implementation direction

1. Separate these concepts in the storage-integrity model:
   • openable;
   • table_readable;
   • structurally_checked;
   • structurally_valid.
2. Reserve storage_state: "ok" for a passed structural check or a still-valid cached attestation.
3. Use unchecked / unknown when the only probe was db_open.
4. Include:
   • attestation_source (live, cached, none);
   • attested_db_fingerprint;
   • attested_at;
   • check_depth (open, quick_check, integrity_check).
5. Keep archive_readability: "readable" if bounded reads succeed; that field is useful and is not contradicted by an autoindex-only failure.
6. Ensure top-level healthy cannot be true when structural integrity is known bad. For merely unchecked state, expose a non-alarming but honest qualification.

Regression test / acceptance criteria

Create a fixture where:

• the table b-tree can be scanned;
• a secondary/unique index has a missing entry or wrong entry count;
• PRAGMA quick_check fails;
• opening the DB and counting the table still succeed.

Assert:

1. cass status --json does not report storage_state: "ok" based only on db_open.
2. source_of_truth_risk is not none when corruption is known.
3. checks_attempted and checks_not_attempted make probe coverage explicit.
4. A cached passed attestation is only reused when its database fingerprint still matches.
5. The output remains bounded and machine-readable; this report does not request an unbounded integrity scan on every status call.
6. Search readiness and storage integrity remain separate fields, so a usable published lexical index can coexist with a degraded canonical-store verdict.

Related issues

• v0.6.19: abort checkpoint logs success after SQLITE_BUSY; DB remains malformed with 1.1G WAL #321 previously showed a malformed database while cass status --json could still report healthy/readable after a stranded WAL. That issue fixed the checkpoint/finalize cause, but the status truth-model gap remains.
• v0.6.22/main regression: bounded doctor busy-spins at ~9.5 GiB RSS and emits zero JSON past external timeout #330 tracks the separate problem that the deeper doctor surface can exceed its budget and emit no JSON.

[Dicklesworthstone]

Fixed in 57848c7. cass status --json and lightweight search readiness now report storage integrity as unchecked unless they can reuse a fingerprint-matching deep-probe attestation; stale DB/WAL metadata invalidates the cache, cached failures gate top-level health, and cass doctor reports live probe provenance. Ordinary doctor check remains strictly read-only: live evidence capture is separated from persistence, and a regression asserts no cache file is created. Exact staged proof: cargo test --lib attestation (4/4), cargo check --all-targets, cargo clippy --all-targets -- -D warnings, cargo fmt --check, and focused UBS all passed through RCH/UBS.