test: add a self-contained MAGSAC++ benchmark suite (synthetic + real datasets)

[f-dy]

test: add a self-contained MAGSAC++ benchmark suite (synthetic + real datasets)

What

A benchmarks/ directory with accuracy benchmarks for pymagsac, plus an A/B
mode to compare two builds on identical data (handy whenever the scoring or
weighting is changed). Baselined on master; entirely additive (no changes to
the library or build).

Estimators (resolved by name in pymagsac at runtime, so unavailable ones
are detected and skipped): fundamental (DoF 4), homography (DoF 2), line2d
(DoF 1) — all on master — and plane3d (DoF 1, findPlane3D), which is not on
master but present in builds that add it. essential (findEssentialMatrix) is
also covered for real data — matching examples/cpp_example.cpp, which has no GT
for essential, it reports the estimated-E inlier ratio + time. --task all runs
whatever the build provides. This lets the same command benchmark the upstream
master build and an improved build (e.g. one adding findPlane3D) and
--compare them directly.

--no-mpp selects MAGSAC (fixed-σ baseline) instead of MAGSAC++, so a compare of
the two is a MAGSAC-vs-MAGSAC++ A/B; the synthetic sweep includes hard regimes
(up to 70% outliers) where the difference is most visible. For each problem the
suite reports — over the ground-truth inliers — err, the RMSE of the
estimated model's residual (the same metric as the repo's examples/cpp_example.cpp,
sqrt(mean(squaredResidual over GT inliers))), the inlier-mask F1, and the
no-model failure fraction (none).

Backends, metrics, sweeps

• --backend: magsac++ (default), magsac (the in-repo baseline), py:MODULE (any pymagsac-compatible
  pure-python module, e.g. py:garfield_sfm.magsac, for A/B vs the C++ binding — use --seed), or
  cv2:NAME — OpenCV find{FundamentalMat,Homography} with method=cv2.NAME
  (cv2:RANSAC, cv2:LMEDS, cv2:USAC_MAGSAC, …), reproducing the paper's
  RANSAC/LMedS/USAC baselines. --compare a baseline CSV against magsac++.
• Metrics: besides the inlier residual, synthetic runs report gt_err — the
  estimate vs the GT model: SGD (symmetric epipolar distance) for F,
  reprojection for H, normal angle for line/plane (the paper's metrics) — plus
  time_ms.
• --sweep-sigma: threshold-sensitivity sweep (the paper's key plot — MAGSAC++
  stays accurate across a wide threshold range while fixed-threshold baselines
  degrade).
• --coherent + --sampler 2: spatially-clustered inliers + P-NAPSAC, to
  exercise the sampler advantage (lower time_ms).

Layout

• synthetic.py — ground-truth generators + metrics. The synthetic GT model is
  the reference, so the metrics are exact (no external oracle).
• datasets.py — real-dataset loaders:
  • pre-computed correspondences (no OpenCV): the repo's own
    <scene>_pts.txt (annotated correspondences + label, the cpp_example.cpp
    format — kusvod2/AdelaideRMF/Multi-H for F, homogr/EVD for H); essential
    <scene>_pts.txt (count + x1 y1 x2 y2, no labels) + <scene>1.K/<scene>2.K
    intrinsics; AdelaideRMF / kusvod2 *.mat (data + label); and homogr / EVD /
    Oxford <stem>.corr + <stem>.H. Discovery is filtered by --task (a mixed
    folder won't cross-load) and works out-of-the-box on the bundled data/
    (47 fundamental, 16 homography, 1 essential scene).
  • image pairs (optional, OpenCV): <stem>A/<stem>B + <stem>_model.txt
    (3×3 GT H); SIFT + Lowe ratio test build the matches — the same recipe as the
    example notebooks. Skipped with a hint if OpenCV is absent.
• run_benchmark.py — unified runner: --task, synthetic sweep / --data DIR /
  --compare A.csv B.csv.
• test_datasets.py — loader unit tests on tiny synthetic fixtures (no download).
• reproduce_paper.py — renders the paper's Table 1 with its published reference rows + our reproduced rows, and the CDF (Fig 3/4) + error-vs-threshold (Fig 5) figures with the paper's legend (matplotlib-gated). Runs on the bundled data/homography directly (top-level=homogr, extremeview/=EVD); --sigma defaults to cpp_example.cpp's σ_max (homography 50, fundamental 5). The H reproduction matches the paper — MAGSAC++ homogr 1.5px (paper 1.3), EVD 5.6px (12.6); LMedS EVD ~94px vs 89.9 (LMedS fails on EVD in both); homogr median is flat at 1.3–1.7px across σ_max 3→100 (the paper's threshold-insensitivity).

Usage

python run_benchmark.py --task all --quick
python run_benchmark.py --task homography --data /path/to/homogr --gt-thresh 3
# A/B upstream master vs an improved build on identical data:
PYTHONPATH=/build_master   python run_benchmark.py --task all --out master.csv
PYTHONPATH=/build_improved python run_benchmark.py --task all --out improved.csv
python run_benchmark.py --compare master.csv improved.csv

Dependencies

numpy (synthetic), scipy (.mat loaders), opencv-python (image-pair loader
only). Datasets are downloaded separately (URLs in benchmarks/README.md); none
are vendored.

Note: a gcransac counterpart

This suite drives pymagsac. An analogous benchmark could be added to
graph-cut-ransac driving pygcransac (same estimators), reusing these
generators and loaders, to additionally cover gcransac's own scoring path. Happy
to open that as a companion PR if useful.