Skip to content

Add inno comply: GDPR deletion verification and data-residue scanning - #211

Merged
ringo380 merged 2 commits into
masterfrom
feat/inno-comply-gdpr
Jul 18, 2026
Merged

Add inno comply: GDPR deletion verification and data-residue scanning#211
ringo380 merged 2 commits into
masterfrom
feat/inno-comply-gdpr

Conversation

@ringo380

Copy link
Copy Markdown
Owner

Implements epic #165 (GDPR & Compliance Verification) - the last unbuilt piece of the v5.0-v5.2 roadmap. Epics 8/9/11 (undelete, timeline, binlog correlation) already shipped.

inno comply is the inverse of inno undelete: instead of recovering data that lingers, it verifies a value has been purged from every InnoDB-retained location in a file and reports residue where it remains.

What's included

  • --verify-deleted --where col=value - decode-and-compare over live, delete-marked, free-list, and undo DEL_MARK records. --thorough adds a raw byte sweep of slack space using the value's InnoDB-encoded form.
  • --scan-residue --pattern <text|hex:..> - literal byte-pattern sweep across all page regions (record heap, free space, header/trailer), reporting page/type/offset/region with hex context.
  • --encryption-audit - encrypted vs plaintext page report and key availability.
  • inno audit --compliance --pattern - directory-wide residue scan across every tablespace, in parallel.
  • Docs - comply CLI reference + a GDPR-verification walkthrough guide, wired into SUMMARY, the CLI overview, and the sitemap.

Design notes

  • Reuses the existing scan surface - scan_free_list_records, scan_undo_for_deletes, decode_page_records, walk_lob/record walkers - so no binary-format parsing was rewritten. Net-new code is the residue/verify orchestration, the value encoder, and the CLI.
  • Honest scope - the scan covers only the files passed in, not the OS page cache, replicas, or other backups, and reports byte/record-level residue rather than certifying legal compliance. The docs and --help state this plainly.
  • No new dependency - patterns are literal (UTF-8 or hex:), so the core library stays regex-free and WASM-lean.

Testing

  • 30 new tests (13 unit + 17 integration) against a real MySQL 9.0 fixture with a known value.
  • Both scan paths mutation-checked: breaking value_matches flips exactly the "present" verify tests; breaking find_all flips exactly the scan tests. Present/absent pairs are the guards.
  • cargo fmt --check, wasm target check, and mdbook build/test all pass. New code is clippy-clean.

Closes #175, #176, #177, #182, #183.

Note: a pre-existing completions_test stack overflow and some newer-clippy-lint errors in unrelated files (simulate.rs, schema.rs, recover.rs, fts.rs) reproduce on master and are untouched here.

…ning

Adds the `comply` subcommand, the inverse of `undelete`: instead of recovering
data that lingers, it verifies a value has been purged from every InnoDB-retained
location in a file and reports residue where it remains.

- verify-deleted: decode-and-compare over live, delete-marked, free-list, and
  undo DEL_MARK records; --thorough adds a raw byte sweep of slack space
- scan-residue: literal byte-pattern sweep across all page regions
- encryption-audit: encrypted vs plaintext page report and key availability
- audit --compliance --pattern: directory-wide residue scan
- docs: comply CLI reference + GDPR verification walkthrough

The scan covers only the files passed in - not the OS page cache, replicas, or
other backups - and reports byte/record-level residue rather than certifying
legal compliance. Patterns are literal (UTF-8 or hex:), keeping the core library
free of a regex dependency and WASM-lean.

Closes #175, #176, #177, #182, #183
@ringo380

Copy link
Copy Markdown
Owner Author

Deep review (auto-fix)

Applied 6 fixes to the working tree (uncommitted - review with git diff, then commit and push):

# Fix Reason Location
1 hex: pattern with non-ASCII input panicked the whole process (&hex[i..i+2] sliced across a UTF-8 char boundary); now validated as ASCII hex digits and returns a clean Argument error bug (crash) compliance.rs#L57-L65
2 --verify-deleted --where col= (empty value) reported every NULL row as residue and flipped fully_purged to false; empty value is now rejected bug comply.rs#L100-L105
3 --thorough relabeled every raw byte-pass hit raw_slack, so a live-record match was reported as unreferenced slack; now tagged raw_<region> (e.g. raw_record_heap) from the real region bug / comment compliance.rs#L525-L531
4 audit --compliance --csv wrote the free-form error field without csv_escape; a comma in an error message corrupted the row (health path already escapes/skips) bug audit.rs#L758-L761
5 Em-dashes in user-visible CLI output, rustdoc comments, and the published mdBook guide replaced with - CLAUDE.md comply.rs#L165-L169
6 comply.md opening overclaimed "purged from every InnoDB-retained location" (the default pass only walks the clustered index, not secondary indexes); reworded to its real scope, and the raw_slack docs corrected to match fix #3 comment comply.md#L1-L4

Verified locally: cargo fmt --check clean, cargo build --release OK, all compliance/comply/audit tests pass (the relabeled verify_thorough_adds_raw_pass included), the panic input hex: non-ASCII now exits 1 not 101, and the changed files are clippy-clean.

Could not auto-fix 2 issues (left for manual review)
  • --encryption-audit reports key_available from encryption_info().is_some(), which is identical to is_encrypted() - so it says "key available: yes" even with no --keyring. The honest signal is decryption_ctx.is_some(), which has no public getter. Fix needs a small accessor on src/innodb/tablespace.rs (outside this PR's files) plus a decision on the intended semantics - skipped rather than guess.
  • audit --compliance --csv on an empty data directory prints the plain-text "no files" message instead of a CSV header. This is the pre-existing shared empty-files block (same gap PR Complete remaining v3.2 issues: depth, prometheus, watch events, web audit #118 fixed once for --prometheus); it is not on lines this PR added, so it's out of scope for an auto-fix here.

- Fixes are uncommitted in the author's working tree. Review before merging.

- Reject non-ASCII hex: patterns instead of panicking on a non-char-boundary slice
- Reject an empty --where value (would match every NULL row and report NOT purged)
- Tag --thorough raw hits with their real page region (raw_<region>) instead of
  always labeling them raw_slack, which mislabeled live-record matches as slack
- Escape the error field in audit --compliance CSV output
- Replace em-dashes with hyphens in CLI output, doc comments, and the guide
- Scope the comply.md summary to the clustered index rather than "every location"
@ringo380
ringo380 merged commit 3cdde97 into master Jul 18, 2026
6 of 12 checks passed
@ringo380
ringo380 deleted the feat/inno-comply-gdpr branch July 18, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data residue scanner — detect if specific column values persist anywhere in tablespace pages

1 participant