pytest-llm-report

Human-friendly pytest test reports with optional LLM annotations.

Features

• Test inventory with status (pass/fail/skip/xfail) and duration
• Per-test covered files and line ranges (using pytest-cov + coverage contexts)
• Per-file source coverage summary (covered/missed/percentage)
• Optional LLM-generated annotations:
  • Scenario: What the test verifies
  • Why needed: What regression/bug it prevents
  • Key assertions: Critical checks performed
  • Smart Context: Automatically fetches model context limits (Gemini, Ollama, LiteLLM) to maximize relevant code included in prompts without hitting token limits.
  • Compact Context: Automatically strips docstrings and optimizes whitespace to reduce token consumption (enabled by default).
  • Token Usage Reporting: Tracks and displays input, output, and total token usage for each annotation and the entire test run.
• HTML and JSON output formats
• Dark mode support (auto-detects system preference)
• Optional PDF generation
• Detailed Versioning: Reports include exact plugin and repository versions (plus Git SHA/dirty status) for reproducibility
• Aggregation across multiple test runs (see Aggregation)

Installation

pip install pytest-llm-report

Or with uv:

uv add pytest-llm-report

Optional Dependencies

Install additional features as needed:

# Ollama provider
pip install pytest-llm-report[ollama]

# Gemini provider
pip install pytest-llm-report[gemini]

# LiteLLM provider (supports OpenAI, Anthropic, etc.)
pip install pytest-llm-report[litellm]

# PDF generation
pip install pytest-llm-report[pdf]

# All features
pip install pytest-llm-report[all]

With uv:

uv add "pytest-llm-report[ollama]"  # or gemini, litellm, pdf, all

Quick Start

Run pytest with coverage contexts enabled:

pytest --cov=your_package --cov-context=test --llm-report=report.html

Configuration

Configure via pyproject.toml:

[tool.pytest_llm_report]
provider = "none"  # "none", "ollama", "litellm", or "gemini"
report_json = "reports/pytest_llm_report.json"

Documentation

📖 Full documentation: palakpsheth.github.io/pytest-llm-report

Requirements

• Python 3.11+
• pytest >= 7.0.0
• pytest-cov >= 4.0.0

Contributing

Contributions are welcome! Please see:

• Contributing Guide - Development setup and guidelines
• Code of Conduct - Community guidelines
• Security Policy - Reporting vulnerabilities
• Changelog - Version history

License

MIT