LLM Fingerprinting System

A black-box fingerprinting system that identifies the underlying LLM model family (GPT, LLaMA, Mistral, etc.) by analysing response patterns across 31 carefully selected prompts. The system can identify fine-tuned models as well, tracing them back to their foundational base model.

Note: Check config.py to see all identifiable model families.

A pre-trained classifier is bundled with the package in the model/ directory.

---

How It Works

Fingerprinting runs in three sequential layers:

1. 31 prompts across 3 layers (discriminative → behavioral → stylistic):

   • Discriminative (11): Identity, knowledge cutoff, architecture, reasoning — most separating power
   • Behavioral (7): Safety boundaries, jailbreak resistance, honesty, policy handling
   • Stylistic (13): Formatting, creativity, constraint following, default voice

2. Feature extraction per response: 384-dim sentence embeddings + 12 linguistic features + 6 behavioral features = 402 dims per layer, 1206 dims total

3. Embedding rebalancing: Per-layer PCA compresses 384-dim embeddings to 64 dims → 246-dim working space

4. Ensemble classification: Random Forest (45%) + SVM (45%) + MLP (10%)

5. Two-stage identification: Ensemble → model family, Template classifier → specific model version

6. Early stopping: After each layer the classifier checks confidence — if it exceeds the threshold (default 0.95) the remaining layers are skipped, saving API calls.

---

Supported Backends

Backend	Description	API Key Required
ollama	Local Ollama instance	❌ No
ollama-cloud	Ollama Cloud API	✅ OLLAMA_CLOUD_API_KEY
openai	OpenAI API (or compatible)	✅ OPENAI_API_KEY
gemini	Gemini API	✅ GEMINI_API_KEY
deepseek	DeepSeek API	✅ DEEPSEEK_API_KEY
grok	xAI Grok API	✅ GROK_API_KEY
claude	Anthropic Claude API	✅ ANTHROPIC_API_KEY
custom	Any HTTP-based LLM API	✅ Optional

About the Custom Backend

The custom backend is the most flexible option — use it with:

• Proprietary LLM APIs not natively supported
• Self-hosted LLMs behind HTTP endpoints
• API proxies and gateways
• Any HTTP-based LLM service

All you need is an HTTP request template file. See examples in ./example/.

If you are integrating through the provider factory (create_provider), use provider="custom" and pass at least request_file. Optional kwargs include api_key and response_path for extracting text from non-standard JSON payloads.

---

Installation

From PyPI

# Core package
pip install llm-fingerprinter

# With OpenAI support
pip install llm-fingerprinter[openai]

# With Gemini support
pip install llm-fingerprinter[gemini]

# With Claude support
pip install llm-fingerprinter[claude]

# With all backends
pip install llm-fingerprinter[all]

Quick Start

1. Identify a Model (Pre-trained Classifier)

# Local Ollama
llm-fingerprinter identify -b ollama --model llama3.2

# OpenAI
export OPENAI_API_KEY="your-key"
llm-fingerprinter identify -b openai --model gpt-4o-mini

# Grok
export GROK_API_KEY="your-key"
llm-fingerprinter identify -b grok --model grok-3-mini

# Custom endpoint
llm-fingerprinter identify -b custom -r ./custom_request.txt

2. Plot Training Fingerprints

Generate 2D semantic/stylistic PCA projections from saved training fingerprints.

# First collect training fingerprints with simulate, then:
llm-fingerprinter plot --input fingerprints/training --output plots/fingerprint_projection.png

The left panel uses the semantic embedding blocks from each fingerprint. The right panel uses the linguistic and behavioral feature blocks.

3. Compare Two Models or Endpoints

Fingerprint two targets with the same prompt suite and inspect whether they identify as the same family, how similar their complete and per-layer fingerprints are, and where classifier probabilities disagree.

llm-fingerprinter compare \
  --left-backend openai --left-model gpt-4o-mini \
  --right-backend ollama --right-model llama3.2 \
  --output comparison.json

Use --left-endpoint or --right-endpoint to compare compatible endpoints, and the matching --*-request-file option for custom HTTP backends.

4. Batch Identify Models

Run a portable JSON configuration to identify multiple models and print a confidence-ranked leaderboard. API keys should remain in environment variables; use api_key_env only when a target needs a non-default key variable.

{
  "models": [
    {"name": "GPT mini", "backend": "openai", "model": "gpt-4o-mini"},
    {"name": "Local Llama", "backend": "ollama", "model": "llama3.2"},
    {
      "name": "Private endpoint",
      "backend": "custom",
      "model": "private-model",
      "request_file": "example/openai_request.txt",
      "api_key_env": "PRIVATE_LLM_API_KEY"
    }
  ]
}

llm-fingerprinter batch-identify --config models.json --output leaderboard.json

Start with the included example/batch_models.json and adapt it to your endpoints.

Each entry requires model; name, endpoint, repeats, request_file, and api_key_env are optional. Relative request files resolve relative to the configuration file. Failed targets are shown in the leaderboard without stopping the rest of the batch.

5. Train Your Own Classifier

# Step 1: Generate training fingerprints for each family
#         Temperature is automatically varied across simulations for diversity
llm-fingerprinter simulate -b ollama --model llama3.2 --family llama --num-sims 5
llm-fingerprinter simulate -b openai --model gpt-4o-mini --family gpt --num-sims 5
llm-fingerprinter simulate -b grok --model grok-3-mini --family grok --num-sims 5

# Step 2: Train the ensemble classifier
llm-fingerprinter train

# Step 3: Build template classifiers (for two-stage identification)
llm-fingerprinter build-templates
llm-fingerprinter build-model-templates

# Step 4: Identify unknown models
llm-fingerprinter identify -b ollama --model some-unknown-model

---

build-templates — Build Family Template Classifier

Compute per-family mean vectors from training fingerprints for the open-set template classifier. Run after train.

llm-fingerprinter build-templates

The template classifier uses cosine distance to nearest mean — it doesn't require retraining when adding new families.

---

build-model-templates — Build Model-Level Templates

Build templates at the specific model version level (e.g. gpt-4o-mini vs gpt-4.1) for two-stage identification.

llm-fingerprinter build-model-templates

Requires fingerprints that contain model_name in their metadata (all fingerprints generated with simulate on this version do).

---

add-family — Add a New Family Without Retraining

Add a new model family to the template classifier from a few fingerprint samples, without retraining the full ensemble.

llm-fingerprinter add-family --model deepseek-chat --family deepseek --num-sims 3 -b deepseek

Recommended minimum: 3 simulations for a reliable mean template.

---

Environment Variables

Variable	Backend	Description
OLLAMA_CLOUD_API_KEY	ollama-cloud	Ollama Cloud API key
OPENAI_API_KEY	openai	OpenAI API key
GEMINI_API_KEY	gemini	Gemini API key
DEEPSEEK_API_KEY	deepseek	DeepSeek API key
GROK_API_KEY	grok	xAI Grok API key
ANTHROPIC_API_KEY	claude	Anthropic Claude API key
LOG_LEVEL	all	Logging level (DEBUG, INFO, WARNING)
LLM_FINGERPRINTER_DATA	all	Override data directory (fingerprints, model, logs)

---

Acknowledgements

This project builds on litemars/LLM-Fingerprinter. This fork extends the original work with model comparison and batch-identification workflows for evaluating multiple LLM endpoints.

---

License

MIT License