Skip to content

[Feature] KPM/NFT Tracking Port to Rust #20

Description

@kalwalt

feat: Port KPM/NFT tracking layer (FreakMatcher) to Rust

Issue body

Summary

This issue tracks the complete porting of the KPM (Keypoint Matching) / NFT
(Natural Feature Tracking) pipeline from C++ to Rust. KPM is the subsystem
responsible for marker-less image tracking using FREAK binary descriptors and
a Difference-of-Gaussians feature detector.

The work is organized as a Strangler Fig migration: a thin FFI bridge
keeps the existing C++ running while Rust replaces it layer by layer. At every
step, a numerical regression test suite validates that the Rust output matches
the C++ output to within floating-point tolerance.


Background

The C++ source lives in
WebARKitLib under
lib/SRC/KPM/ and lib/SRC/KPM/FreakMatcher/. The Rust port lives in
this repository (crates/kpm/).

What has already been ported (in crates/core/):

Module File Status
ICP pose estimation src/icp.rs ✅ Complete
Matrix math (ARMat) src/math.rs ✅ Complete
Core types (ARParam, ARdouble) src/types.rs ✅ Complete

What this issue covers — the new crates/kpm/ crate:

Layer C++ source Effort
KPM types & structs kpmType.h, kpm.h Low
KPM public API kpmHandle.cpp, kpmResult.cpp Low
Image resize helpers kpmUtil.cpp (genBWImage*) Low
Ref dataset I/O kpmRefDataSet.cpp Low
kpmMatching orchestration kpmMatching.cpp Medium
Math utilities math/indexing.h, math_utils.h Low
Linear algebra & solvers math/linear_algebra.h, linear_solvers.h Medium
Homography estimation homography_estimation/robust_homography.h High
Hough voting + feature matcher matchers/hough_similarity_voting.h Medium
Binary hierarchical clustering matchers/binary_hierarchical_clustering.h High
FREAK descriptor matchers/freak.h, keyframe.h High
DoG detector + Gaussian pyramid detectors/DoG_scale_invariant_detector.* High
VisualDatabase + facade matchers/visual_database.*, facade/ High

Key dependency on Eigen: Eigen is used in exactly one place —
IncrementalHomographyFromLieWeights in robust_homography.h —
for a 3×3 matrix exponential. This will be replaced with either
nalgebra's matrix-exp feature or a hand-written Padé approximation.
The original authors left a // TODO: remove Eigen comment there.


Architecture

// note this was refactored: Now we have only core and wasm crate.
// kpm is inside core -> core/kpm
crates/kpm/
├── build.rs              # compiles C++ FreakMatcher as static lib (FFI phase)
├── src/
│   ├── lib.rs
│   ├── kpm_c_api.h       # thin C wrapper over C++ KPM (FFI phase)
│   ├── kpm_c_api.cpp     # implementation of C wrapper
│   ├── kpm_ffi.rs        # bindgen-generated bindings (FFI phase)
│   ├── types.rs          # KpmHandle, KpmRefDataSet, FreakFeature, etc.
│   ├── backend.rs        # FreakMatcherBackend trait
│   ├── cpp_backend.rs    # CppFreakMatcher: implements backend via FFI
│   ├── handle.rs         # KpmHandle struct (orchestration)
│   ├── ref_data_set.rs   # KpmRefDataSet: generate, save, load, merge
│   ├── matching.rs       # kpm_matching() + kpm_util_get_pose_binary()
│   └── freak/            # (Milestone 3+) pure Rust FreakMatcher
│       ├── mod.rs
│       ├── math.rs
│       ├── homography.rs
│       ├── hough.rs
│       ├── clustering.rs
│       ├── descriptor.rs
│       ├── detector.rs
│       └── database.rs
└── tests/
    └── regression.rs     # numerical regression vs C++ baseline

Cargo features

Feature Purpose
ffi-backend (default) Enables C++ FreakMatcher via FFI
dual-mode Runs Rust and C++ in parallel and logs divergences

When ffi-backend is disabled and the pure Rust FreakMatcher is complete,
the C++ dependency is dropped entirely.


How to contribute

  1. Pick a task from the milestone that interests you.
  2. Each task has a corresponding prompt in CLAUDE_CODE_PLAN.md
    at the repo root — feel free to use it with Claude Code.
  3. Every new function must have a unit test and must not break
    cargo test -p kpm.
  4. Numerical functions must pass the regression tests in
    crates/kpm/tests/regression.rs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions