Skip to content

fix(highlighter): preserve bounded syntax context - #144

Merged
owainlewis merged 1 commit into
mainfrom
codex/issues-130-131-highlighter
Jul 28, 2026
Merged

fix(highlighter): preserve bounded syntax context#144
owainlewis merged 1 commit into
mainfrom
codex/issues-130-131-highlighter

Conversation

@owainlewis

Copy link
Copy Markdown
Owner

Summary

  • Keep line zero plus eight least-recently-used window-boundary syntax checkpoints so warmed distant Rust and Markdown viewports remain bounded across backward and forward jumps.
  • Preserve block-quoted backtick and tilde fence context, including injected Rust, synthetic seeds, contents, visible closers, and container-prefix edits.
  • Stop the [performance] Bound Rust syntax seed for deeply nested block comments #124 conservative Rust overflow fallback at truncated-line barriers, Markdown fence boundaries, and quoted-container loss while preserving the exact 64-level nested-comment bound.
  • Add the combined design and verification record in docs/issues/130-131-plan.md.

Why

The existing checkpoint insertion discarded every later checkpoint during a backward jump, so alternating between distant locations repeatedly rescanned file-depth prefixes. The Markdown checkpoint scanner also missed fences nested under > containers, causing deep injected Rust context loss. After #124, overflow fallback bypassed truncated-line barriers and could keep visible code styled as a comment.

These fixes stay inside the bounded checkpoint/window design. They retain only deterministic window anchors, carry the minimal quote-depth context needed by the scanner, and split conservative overflow at the same boundaries used by normal context recovery.

Test plan

Verified on exact base b3ef75d0ea74b5a13e022a34699396fb878e03ed and head b0040dbe104d6e0d60af0e97ccbba47c007a4a95:

  • cargo test highlighter::tests:: -- --nocapture: 44 passed.
  • cargo test performance:: -- --ignored --nocapture --test-threads=1: all 4 passed; deep highlighting completed in 2.381s initially and 22.473ms after edit.
  • cargo fmt --check.
  • git diff --check and staged diff validation.
  • cargo test --all-targets: 308 passed, 4 documented performance tests ignored, and all 5 signal cleanup tests passed.
  • cargo clippy --all-targets --all-features -- -D warnings.
  • cargo build --all-targets.
  • Interactive PTY smoke: opened highlighted Markdown, exited with C-x C-c, and reused the restored shell (CORTEX_SHELL_OK).
  • Applied the named improve workflow to clarify checkpoint and overflow-boundary invariants without changing behavior.
  • Initial independent review found quoted overflow did not stop at container loss. The implementation and a quoted 65-opener barrier/container-edit regression fixed it. Fresh final independent review approved the unchanged highlighter patch against exact base b3ef75d with no findings.

Risks

The Markdown checkpoint scanner intentionally supports block-quote containers without becoming a complete CommonMark parser. A quoted fence requires its block-quote container on opener, contents, and closer. Losing that container ends the bounded fence context.

Rust block-comment nesting above 64 levels remains conservative by design. Normal Rust resumes at a truncated-line barrier. Fenced Markdown also resumes at its real closer, a truncated-line barrier, or loss of the required quote container.

Checkpoint memory remains capped at line zero plus eight recent anchors. An edit retains checkpoints at or before its first changed line and invalidates later contexts that may be stale.

Related issue

Closes #130
Closes #131
Closes #142

@owainlewis

Copy link
Copy Markdown
Owner Author

@codex review

@owainlewis
owainlewis merged commit 0e9ad92 into main Jul 28, 2026
1 check passed
@owainlewis
owainlewis deleted the codex/issues-130-131-highlighter branch July 28, 2026 22:41
@owainlewis

Copy link
Copy Markdown
Owner Author

Merged-main verification complete.

Merge commit: 0e9ad9293734a0384fb46e8e839cd4d891b74062 (current origin/main).

Post-merge CI: https://github.com/owainlewis/cortex/actions/runs/30405444201

Rust checks passed in 1m27s, including security workflow policy, formatting, strict Clippy, tests, and release build. Issues #130, #131, and #142 are closed. The original #141 review thread is resolved with commit/PR and focused regression evidence.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e9ad92937

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/highlighter.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant