[TEST] Neo4j conformance in CI: ratcheted three-route differential against pinned Neo4j, allocation gate for hot paths, and Neo4j TestKit for driver-level conformance

[mm0nst3r]

Summary

Divergences from Neo4j keep being found by manual differential runs after they reach main, not by CI. Examples from the last two days:

• [BUG] Text that changes length when upper-cased (ı) shifts keyword positions: RETURN, SKIP/LIMIT, CALL { }, CASE and UNWIND panic, fail or return wrong columns #748: non-ASCII text shifted keyword positions;
• [BUG] A statement TERMINATE TRANSACTIONS stops isn't Transaction.Terminated: a write fails as SyntaxError "context canceled" (since #747), a read returns its rows #751: TERMINATE returned the wrong status;
• USE db; … answered DatabaseNotFound instead of a SyntaxError (caught in fix(cypher): USE runs on the database it names; one USE grammar; composite graph references (#738) #749 before merge);
• fix(cypher): USE runs on the database it names; one USE grammar; composite graph references (#738) #749's per-row allocation regression ([PERF] Overhead added by recent convergence fixes: RETURN re-parse, membership ToUpper, compiled-WHERE wrappers, pipeline keyword scanner, pipeline SET allocations #591 section, fixed by perf(cypher): a function context's graph catalog costs no allocation per row (part of #591) #752).

Each was found by comparing NornicDB with the pinned Neo4j over several routes, or by comparing allocations against the previous main. None of these runs is in CI.

This issue proposes three CI additions so that a divergence from Neo4j, or a hot-path allocation regression, fails the PR that introduces it.

What CI has today

.github/workflows/cypher-conformance.yml:

• make cypher-conformance: the openCypher TCK over Bolt, with a ratchet.
• make cypher-differential (scripts/cypher-tck/run-differential.sh): starts the pinned neo4j:5.26.30-community image and checks that a fixed corpus returns the same results as Neo4j (TestFixedDifferentialCorpusMatchesPinnedNeo4j).

The differential covers only statements that already match, and only Bolt auto-commit.

Proposal

1. A ratcheted, three-route differential

• Corpus: the fixed corpus, plus every statement from the reproductions in open and closed issues (the tables and code blocks in issue texts), plus a per-family case sweep (setup + statements).
• Routes: every statement runs through Bolt auto-commit, a Bolt explicit transaction and HTTP /db/<db>/tx/commit, against NornicDB and the pinned Neo4j. Compare columns, rows (unordered when the statement has no ORDER BY), error class and code, and, for writes, the graph state after the statement.
• Ratchet: a checked-in list of known mismatches, one per statement and route, each linked to its issue. CI fails when a statement that matched starts differing, and reports statements that newly match so the list can be trimmed in the same PR.
• Harness robustness: the reset between cases retries a transient error ([BUG] MVCC / storage key lifecycle has no single owner: prune conflicts with concurrent writes and runs through two implementations, DROP DATABASE leaves ~89% of keys (MVCC heads and history included), a recreated ID inherits dropped history, MVCC benchmarks measure nothing (family) #732's race) and counts the retries, so a run is never cut short. A crashed run fails the job instead of comparing partial results.

2. An allocation gate for hot paths

Allocation counts at a fixed iteration count (-benchtime=Nx -benchmem) are deterministic, unlike timings on shared runners. A job builds the base and the head, runs the cypher and storage benchmarks at a fixed count, and fails when any benchmark allocates more per operation than on the base, unless the PR lists it with a reason. Timing comparisons stay out of this job (they need a dedicated runner).

3. Neo4j TestKit (driver-level conformance)

neo4j-drivers/testkit is Neo4j's own harness that runs each official driver (Go, Python, Java, JavaScript, .NET) against a server. Its server-backed test suites, run against NornicDB, exercise Bolt-level behaviour through real drivers: value types, transactions, bookmarks, error classification and retry behaviour. Today these are checked only indirectly through one driver.

• Run it as a separate workflow, nightly and on demand, with Python and Go to start, against NornicDB and the pinned Neo4j side by side.
• Record a baseline of expected failures (tests needing cluster routing or Enterprise features) as a ratchet, the same way as the TCK. Fail only when a test that passed starts failing.

Acceptance

• A PR that changes a statement's result, error code or written graph state relative to Neo4j (for a statement that matched before) fails CI, on whichever route it differs.
• A PR that adds allocations per operation to any cypher or storage benchmark fails CI unless the increase is declared.
• TestKit runs nightly with a committed baseline, and its report lists newly failing tests.

Consequence (of not having this)

• Divergences from Neo4j reach main and are found later by manual runs. Some are silent wrong results or lost writes ([BUG] Text that changes length when upper-cased (ı) shifts keyword positions: RETURN, SKIP/LIMIT, CALL { }, CASE and UNWIND panic, fail or return wrong columns #748 lost whole explicit transactions).
• Hot-path allocation regressions reach main unnoticed (fix(cypher): USE runs on the database it names; one USE grammar; composite graph references (#738) #749 → perf(cypher): a function context's graph catalog costs no allocation per row (part of #591) #752).
• Driver-level incompatibilities (types, bookmarks, retries) are found only when a user's driver hits them.