test(numeric): declarative torch-mlir-style registration harness for per-op numeric tests

[chizamd]

Motivation

Raised by @fhanuman in PR #283 review: the test/numeric/ per-op tests carry a fair amount of repeated scaffolding, and torch-mlir's e2e test framework is a good reference for a more declarative style.

We already have a real framework layer under test/numeric/framework/ (backend ABC, ORT-CPU reference + disk cache with SHA256 drift detection, compare_outputs, make_model_from_nodes). What's missing is the torch-mlir-style declarative registration that sits on top of it.

What to borrow from torch-mlir

• One declaration → both reference and EP run derive from it. A case declares (model builder, seeded input factory, reference, tolerances); the harness runs the EP and computes the reference, no hand-stored expected output.
• @numeric_case + a GLOBAL_NUMERIC_REGISTRY (registration as an import side effect; one generated pytest parametrizes over the registry).
• Seeded input factory so reference and EP see bit-identical inputs.
• (optional) name-keyed xfail set + XPASS-is-failure hygiene.

Key constraint (don't regress this)

The reference must support two modes as first-class:

1. CpuRef — run the same ONNX on ORT CPU (+ existing disk cache). Used by most ops.
2. NumpyRef(fn) — a python callable computes the expected output. Required for the whisper attention tests: ORT CPU fp16 attention softmax drifts to cosine ~0.6, so those tests carry their own fp64 numpy reference and currently bypass run_sample with a copy-pasted _persist helper. The harness must fold this in, not force it back out.

Honest scope note

This is not a "delete 60% of test code" change. The per-op _make_*_model ONNX-graph factories are mostly irreducible (a Conv graph and an Attention graph share almost nothing). The win is removing the repeated glue (input-gen + run/compare + the _persist bypass duplicated across the 3 attention files) and giving one place for tolerances / GPU-dispatch assertion / cache handling — roughly 15–30% per file, concentrated in scaffolding.

Proposed plan (incremental, not big-bang)

1. Add framework/registry.py: NumericCase dataclass + CpuRef/NumpyRef + @numeric_case + a conftest driver that parametrizes over the registry.
2. Port one cache-ref file (test_matmul.py) and one numpy-ref file (one whisper attention test) as proof.
3. Land that, then port the remaining ~15 files incrementally.

Keep run_sample / compare_outputs / make_model_from_nodes unchanged — the harness sits on top of them.

Out of scope

• test/python/ model suite — already spec-driven (ModelSpec / BaseORTTests), different problem shape, leave it.
• test/lit/ — MLIR-native, no Python harness applies.

Open questions

• Name-keyed xfail set (torch-mlir style) vs keeping pytest xfail marks? (Single-backend suite, so the multi-backend pass-set machinery is likely overkill.)
• Adopt "XPASS is a hard failure" CI semantics?

A fuller writeup of the research (current-state line-budget breakdown, torch-mlir mechanics, a registry.py sketch, before/after of the whisper encoder test) is available and can be added to docs/design/ alongside the implementing PR.

Originating review thread: #283 (#283 (comment))