[Feature]: Toploc as validation mechanism

[fedor-konovalenko]

🚀 The feature, motivation and pitch

We propose to add TOPLOC algorithm in order to reduce artifact size. The goal is to study the applicability of TopLoC: will using this algorithm improve validation reliability without increasing storage costs? This involves finding the optimal approach and testing on synthetic and real data.

TOPLOC Overview

TOPLOC is a novel verification method designed to ensure that AI inference providers are using the specific models and prompts they claim without unauthorized changes. By utilizing a locality-sensitive hashing scheme, the system tracks the most significant intermediate activations to detect modifications in model weights, prompts, or numerical precision. A key innovation is the use of polynomial encoding, reducing necessary storage size. This approach remains robust against hardware non-determinism and GPU-specific variations that typically cause minor numerical discrepancies. Empirical testing confirms the method achieves 100% accuracy in identifying tampering while maintaining validation speeds faster than the original inference.

We plan to implement TOPLOC algorithm without affecting other functionality and without influencing the performance of the platform core. During the development process, we plan to adapt the approach from the article described above to the inference of visual language models and integrate the necessary tools into the inference engine.

More details and links to our code with TopLoc will be added later in the discussion

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[aby2s]

A clarification of the ticket description above, based on the offline benchmarks we ran on Qwen3-235B-A22B-Instruct-2507.

Summary

Replace the current top-k-logprobs similarity check with a TOPLOC-style polynomial commitment over top-k hidden-state activations, exposed through the same validation flow. TOPLOC makes artifact size tunable: at the ~40 B/token the current logprobs check already pays, it gives a comparable detection rate; and it can win up to ~10× in artifact size while still maintaining a good-enough detection rate.

Motivation

The current logprobs-similarity check works well — on the fraud scenarios we tested it gives comparable or even slightly better detection rates than TOPLOC at any matching cost. The limitation is that it is hard to tune: artifact size and detection quality are tied to a single mechanism (top-N logprobs + a similarity heuristic), with no clean way to either shrink the
artifact or push detection up.

TOPLOC (arXiv:2501.16007) gives the same task two explicit knobs:

• k (top-k of the hidden state) sets per-proof size.
• decode_batch_size (tokens per proof) sets how many tokens one proof
  covers.

Together they trace an explicit cost / detection-quality frontier that the protocol can pick a point on per request class, instead of a fixed operating point baked into the validation mechanism.

In our offline benchmark on Qwen3-235B-A22B-Instruct-2507 (1100 multilingual prompts, FP8 weights, FP8 KV cache, A100 ↔ H100 cross-architecture positives, INT4-served-as-FP8 fraud as negative), the frontier at FPR ≤ 0.5% on honest cross-arch pairs is:

operating point	k	bs	bytes/token	FNR vs INT4 fraud
strict reference	512	8	129.6	0.0%
strict reference	512	16	65.7	0.0%
match-current size	512	32	33.7	<= 1% (0.0% at 1% FPR)
2x smaller	64	8	16.4	4.5%
2x smaller	256	32	16.5	4.0%
10x smaller	64	32	4.2	~15%

Even at ~10× smaller artifacts, the false-negative rate stays low enough to detect fraud with high confidence.

And because the size is a knob rather than a fixed cost, it can later be made adaptive — varied per model type, prompt length, request importance/risk class, or any other policy the network wants — giving the protocol a flexibility the current fixed-shape logprobs check does not have.

On this dataset we can't directly compare TOPLOC's FPR against the logprobs check. The few prompts that fail produce very different outputs on A100 vs H100, so the model itself is behaving differently across architectures and it's not a TOPLOC issue. We'd need more experiments to pin this down. Details in the research summary.

Expected outcome

• Instrument the GPU model runner to compute and emit top-k hidden-state proofs alongside (or instead of) the existing logprobs path.
• /v1/validate returns a TOPLOC verdict alongside (or replacing) the current similarity score. Hidden states are never exposed to the client.
• Default thresholds calibrated for the production model and at least the A100 ↔ H100 and H100 ↔ H100 hardware pairs.
• A documented size / detection-quality curve so the protocol can pick a point and revisit it as new hardware joins the network.

Experimental results

Full report with figures
.

[aby2s]

Hey! We ran a few more experiments on the most recent version of vLLM combined with logprob collection, and the updated report is here: gonka_research_summary.md.

A few highlights from the recollection:

• The gonka logprobs validator yields ~20% false positives on this honest A100→H100 cross-arch pair at the default 0.99 threshold (875/1100 pass, mean similarity 0.9922). Admitting 100% of honest cross-arch FP8 pairs needs the threshold lowered to ~0.98. Long sequences (>2000 tok) sit just below 0.99 almost deterministically.
• The two persistent A100 outliers (605/692) don't reproduce on the newer vLLM — they now sit at average ranks 425/1100 and 326/1100 across (k, bs) configs. Without them dragging the threshold up, FPR = 0% becomes a usable regime for TOPLOC.
• The seq-length-aware threshold was redone with a better linear-regression fit (per-sample mean mant_err vs log₁₀(seq_len), lifted to hit exactly the FPR budget). The line tilts downward with sequence length and buys a few percentage points of FNR for free in the cheap configs.
• Same thresholds, same distribution of mantissa errors as the original run. Honest cross-arch mean mant_err is uniformly ~0.15–0.18 higher on the recollection (e.g. k=128/bs=128: 3.12 → 3.29) — small numerical shifts as expected from PyTorch kernel and sampler-path differences. Heatmap structure is unchanged.

One bonus observation in the report: the two validators are statistically uncorrelated at the per-sample level (Pearson r ∈ [-0.01, +0.18], Spearman ρ ∈ [-0.11, +0.08] across (k, bs) configs), since gonka measures post-LM-head logprob-distribution drift and TOPLOC measures pre-LM-head hidden-state-coordinate drift — a fraudulent run would need to game both signals simultaneously.

[fedor-konovalenko]

A small clarification regarding the comparison of results with the original paper.

The original paper demonstrated near-100% algorithm accuracy. However, the authors experimented with small models - LLama-8B. Our results for small models were indeed consistent with the original paper's results, but for larger models (235b), accuracy was lower.

Therefore, in my opinion, the proposed TOPLOC algorithm, while potentially interesting, is not the first priority for implementation in upcoming releases.