interp-engine is an interpretability engine that is fast, standardized (34 'points'/addresses across architectures), and easy to use and debug. It powers all of Neuronpedia's inference and is checked for accuracy against HF Transformers and other engines.
This repo contains:
validator/, which compares/validates it against TransformerLens, and nnsight/nnterp on real architectures.visualizer-web/, a "cheat sheet" hosted at interp-engine.org of each 'point' (egresid_post.16), standardized across architectures.gpu-sizer/, which finds the GPU/config you need to fit an interp-engine model (and what performance you'll get), to avoid OOMing while working. docs | API
pip install 'interp-engine[vllm]' # preferred install: includes vLLM support (CUDA required)
pip install interp-engine # eager backend onlyfrom interp_engine import Address, load_model, run_with_cache
# VLLM (default): low VRAM, medium speed, every point, chosen per request
model = load_model("Qwen/Qwen3-8B")
# VLLM-STATIC: high VRAM, high speed, only the points you declare (default resid_post)
# model = load_model("Qwen/Qwen3-8B", backend="vllm-static")
# VLLM-GENERATE: fastest, generation only -- no capture, no steering
# model = load_model("Qwen/Qwen3-8B", backend="vllm-generate")
# EAGER: low VRAM, low speed
# model = load_model("Qwen/Qwen3-8B", backend="eager")
point = Address("resid_post", 10) # or string: "resid_post.10"
cache = run_with_cache(model, model.to_tokens("Hello, world"), [point])
cache[point] # [batch, pos, ...]Add "use interp-engine" to your prompt and let your agent figure it out - everything is fully documented in this repo and open source.
interp-engine supports 34 standardized points ("Addresses") across architectures: every one of them on the eager backend, 28 of them on vLLM. Check interp-engine.org for the "cheat sheet", or SUPPORTED_POINTS.md for a markdown version with the per-backend detail.
vLLM gives interp-engine high throughput via concurrency, and backend="vllm-static" gives even higher throughput at the cost of higher VRAM usage. Every column below is capture-capable.
Measured on NVIDIA B200, bf16, 512-token prompt, 128 new tokens.
One stream (tok/s):
| model | eager | vLLM | vLLM + static taps |
|---|---|---|---|
gemma-2-2b |
75 | 88 (1.2x) | 220 (2.9x) |
qwen3-4b |
60 | 136 (2.3x) | 313 (5.2x) |
llama-3.1-8b |
84 | 163 (1.9x) | 258 (3.1x) |
qwen3.8-27b |
25 | 35 (1.4x) | 63 (2.5x) |
deepseek-v4-flash-0731 |
5.3 | 15 (2.9x) | 132 (25x) |
8 concurrent requests (aggregate tok/s):
| model | eager | vLLM | vLLM + static taps |
|---|---|---|---|
gemma-2-2b |
75 | 675 (9.0x) | 1,436 (19.2x) |
qwen3-4b |
61 | 973 (15.9x) | 2,219 (36x) |
llama-3.1-8b |
81 | 1,187 (14.6x) | 1,755 (22x) |
qwen3.8-27b |
24 | 246 (10.3x) | 408 (17.0x) |
deepseek-v4-flash-0731 |
5.4 | 117 (22x) | 631 (116x) |
backend="vllm-static" is opt-in, and serves only the tap set it declared — static_points="auto" by default, or a list you name. PERFORMANCE.md has how it works and what it trades; benchmarks/results-latest.md has every figure at full precision, including capture, steering and lens latencies; benchmarks/README.md has how the tables above are rounded.
We verify correctness in two main ways:
- A test suite that checks results across several models - what each check is designed to catch is in INTERNALS.md.
- A full
validatorcomparison engine that checks most hook points across 50+ models, at early, middle and late layers - fully reproducible, with detailed results saved in the git repo atvalidator/.
Never OOM again - interp-engine includes gpu-sizer, an intuitive UI which tells you what GPU(s) and configs you need to get the best performance out of your selected model. For example, interp-engine.org/sizer/Qwen/Qwen3.6-27B shows that you can run Qwen3.6-27B with interp-engine's vllm-static backend for max speed while keeping it in a single 80GB A100, and have ~160k tokens for the KV Cache.
- Speed: Get performance without sacrificing correctness.
- Standardization + Verification: Eliminate ambiguity when referring to points, plus a full test suite included.
- Faster Dev / Fewer Tokens Used: You could spend ten million tokens and have your AI write, test, and make production-ready an interpretability engine. Or you could just
pip install interp-engine[vllm].
- Gemma 4 requires transformers 5.14.1 because 5.15 moved
head_dimintoper_layer_configand vLLM's config read dies before a weight loads (vllm#51744). - DeepSeek-V2 on transformers older than 5.15.0 captures a wrong attention temperature — the engine warns at load, and upgrading is the fix (COMPATIBILITY.md).
- DeepSeek-V4 on the eager backend needs two flags outside Hopper. Its FP8 paths reach for
kernels-community/deep-gemm, which declares a9.0abuild, and withkernels>= 0.16.1 the arch refusal escapes transformers' Triton fallback and kills the first forward. Load withTRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1andexperts_implementation="grouped_mm", which transformers already recommends on B200 for an unrelated DeepGEMM accuracy problem. The engine raisesHubKernelUnsupportednaming both; vLLM is unaffected. - MXFP4 checkpoints (gpt-oss) need
interp-engine[quant], which the[vllm]extra does not include; without it transformers dequantizes them to bf16 at roughly 3x the weights, which can turn a model that fits into one that does not.
Per-architecture structural quirks — which points a family serves and why — are not caveats but facts about the architecture, and live in ARCHITECTURE_QUIRKS.md.
Activate the shared git hooks once per clone — they format staged Python, rebuild the generated files, and run CI's static checks before a push. Details in CONTRIBUTING.md.
make hooks # or: git config core.hooksPath .githooksBugs and feature requests belong in issues. For anything else: johnny@neuronpedia.org.
Apache 2.0