Generalize document clustering fingerprints

[Mallets]

Summary

• Generalize document clustering fingerprints from the fixed schema/grouping pair to an ordered list of fingerprint policy hashes.
• Remove the config validation that forced exactly two fingerprint policies and add validated JsonPath handling for raw/tokenized paths and structure exclusions.
• Switch DocIdClusterer to a recursive fingerprint-prefix tree that emits largest groups first at every fingerprint level.

Motivation

This change makes document clustering implement what the configuration already allows expressing: an ordered list of fingerprint policies rather than a fixed (schema, grouping) pair.

Each policy produces an independent hash from its configured fields. These hashes form successive clustering levels:

document fingerprint: [h0, h1, h2, h3]

root
├── h0 = A
│   ├── h1 = X
│   │   ├── h2 = P
│   │   │   ├── h3 = I → [doc 1, doc 8]
│   │   │   └── h3 = J → [doc 3]
│   │   └── h2 = Q
│   │       └── h3 = K → [doc 5, doc 9]
│   └── h1 = Y
│       └── ...
└── h0 = B
    └── ...

At every level, sibling clusters are emitted largest first before recursively ordering their children. Each level can use a different hashing policy, e.g. structure, tokenized message shape, and service name.

The previous implementation collapsed this configuration into exactly two steps. The new implementation preserves every configured step and supports arbitrary fingerprint depth.

Validation

• cargo +nightly fmt --all
• cargo test -p quickwit-indexing docs_clustering::clusterer --lib
• cargo test -p quickwit-indexing docs_clustering --lib
• cargo clippy -p quickwit-indexing --all-features --tests

Benchmark

All benchmarks used 100k synthetic documents. Melem/s denotes millions of documents processed per second.

The main baseline runs the fixed two-level fingerprinter and clusterer. Fingerprints are precomputed for insertion and sorting benchmarks. Sorting receives fully built clusterers through Criterion's untimed setup and measures sorting/finalization, including construction and validation of the final DocIdMapping.

Phase	Scenario	main	Current
Fingerprint calculation	two policies	8.20 Melem/s	8.11 Melem/s
Document insertion	low cardinality	121.86 Melem/s	71.29 Melem/s
Document insertion	high cardinality	38.40 Melem/s	28.55 Melem/s
Cluster sorting/finalization	low cardinality	268.50 Melem/s	605.57 Melem/s
Cluster sorting/finalization	high cardinality	185.37 Melem/s	206.11 Melem/s

The recursive representation makes insertion slower than the specialized fixed two-level implementation, but sorting/finalization is substantially faster for low-cardinality clusters and about 11% faster for high-cardinality clusters. Fingerprint calculation is effectively unchanged.

By default, insertion is limited to 4 MiB per shard, and actual indexing throughput is far below tens of millions of documents per second. The measured insertion rates therefore leave substantial headroom and are unlikely to be the limiting factor. Sorting/finalization time is more performance-critical, and that phase improves with the recursive design.