Mcp.BenchmarkRunner

Run .NET benchmarks from an LLM.
An MCP (Model Context Protocol) server in C# that executes BenchmarkDotNet on demand and returns results & artifacts. Works with MCP Inspector and GitHub Copilot Chat (Tools) in Visual Studio.

Maintained by Aliaksandr Marozka — amarozka.dev

TL;DR – Ask your AI to “run the hash benchmarks,” get back Mean/Alloc deltas and Markdown/JSON/CSV artifacts. Perfect for demoing MCP, perf checks in PRs, or quick regression hunts.

---

✨ Features

• MCP tools
  • run_bench(filter?, job?, exporters?, timeoutSec?) – run benchmarks, get a runId + brief JSON summary
  • get_results(runId) – parsed JSON summary (Mean, Alloc/Op, etc.)
  • list_artifacts(runId) – list all files (JSON / Markdown / CSV / logs)
  • compare_runs(baseRunId, headRunId) – simple deltas (% change for Mean & AllocatedBytes/Op)
• Job presets: Short, Medium, Long
• Exporters: json, fulljson, md, csv
• Clients: MCP Inspector (stdio) and Visual Studio (Copilot Chat Tools)

---

🗂 Project Layout

Mcp.BenchmarkRunner.sln
│
├─ src/
│  ├─ Mcp.BenchmarkRunner.Server/          # MCP server (console)
│  │  ├─ Program.cs                        # host + stdio transport + attribute discovery
│  │  └─ Tools/
│  │     ├─ BenchTools.cs                  # run_bench / get_results / list_artifacts
│  │     └─ CompareTools.cs                # compare_runs
│  │
│  └─ SampleBenchmarks/                    # BenchmarkDotNet executable
│     ├─ Program.cs                        # CLI: --artifacts/--exporters/--job/--filter
│     ├─ Jobs.cs                           # Short/Medium/Long presets
│     └─ HashBench.cs                      # MD5 vs SHA256 demo
│
└─ runs/                                   # output per runId (created at runtime)

Flow: Inspector/Copilot → MCP server (stdio) → spawns dotnet run on SampleBenchmarks → BDN writes artifacts → server returns JSON/paths.

---

🚀 Quickstart

Prerequisites

• .NET SDK 9.0 (or switch both projects to .NET 8.0)
• (Optional) Node.js for MCP Inspector (npx)

Build & sanity check (CLI)

dotnet restore
dotnet build

# Run benchmarks without MCP (sanity)
dotnet run --project src/SampleBenchmarks/SampleBenchmarks.csproj -c Release -- \
  --artifacts runs/manual1 --exporters json,md --job Short --filter *Hash*

# Start the MCP server
dotnet run --project src/Mcp.BenchmarkRunner.Server

Artifacts appear under runs/<runId or manual1>.

---

🧪 MCP Inspector (standalone)

Start Inspector:

npx -y @modelcontextprotocol/inspector
# or:
npm i -g @modelcontextprotocol/inspector
mcp-inspector

Connect (stdio):

• Command: dotnet
• Args: run --project src/Mcp.BenchmarkRunner.Server/Mcp.BenchmarkRunner.Server.csproj

Example payloads:

{ "filter": "*Hash*", "job": "Short", "exporters": "json,md" }

{ "BaseRunId": "<first>", "HeadRunId": "<second>" }

Windows npm ENOENT fix
If npx fails with %APPDATA%\npm not found:
mkdir "%APPDATA%\npm" && npm config set prefix "%APPDATA%\npm" and add %APPDATA%\npm to PATH.

---

🧰 Visual Studio (Copilot Chat Tools)

Option A — .mcp.json in the solution root (recommended)

{
  "servers": {
    "benchrunner": {
      "name": "Benchmark Runner",
      "type": "stdio",
      "command": "dotnet",
      "args": [
        "run",
        "--project",
        "src/Mcp.BenchmarkRunner.Server/Mcp.BenchmarkRunner.Server.csproj"
      ]
    }
  }
}

Option B — global %USERPROFILE%\.mcp.json (use absolute path)

{
  "servers": {
    "benchrunner": {
      "name": "Benchmark Runner",
      "type": "stdio",
      "command": "dotnet",
      "args": [
        "run",
        "--project",
        "C:\\path\\to\\repo\\src\\Mcp.BenchmarkRunner.Server\\Mcp.BenchmarkRunner.Server.csproj"
      ]
    }
  }
}

Use it: View → GitHub Copilot Chat → Tools/Agent → enable Benchmark Runner → run:

• run_bench → grab runId
• get_results / list_artifacts
• compare_runs with two runIds

Logs: View → Output → GitHub Copilot.

---

🧩 Tool Contracts

run_bench

{ "filter": "string?", "job": "Short|Medium|Long", "exporters": "json|fulljson|md|csv", "timeoutSec": 600 }

Returns: { "runId": "string", "artifactsDir": "path", "summary": { ... } }

get_results

{ "runId": "string" }

Returns: parsed JSON exporter object or null.

list_artifacts

{ "runId": "string" }

Returns: list of file paths under runs/<runId>.

compare_runs

{ "BaseRunId": "string", "HeadRunId": "string" }

Returns: rows with Mean/Alloc base/head and %Δ.

---

🛠 Troubleshooting

• restore failed
  • Enable nuget.org source; clear caches:
    dotnet nuget add source https://api.nuget.org/v3/index.json -n nuget.org
dotnet nuget locals all --clear
  • Enable prerelease (MCP SDK is preview) or switch both projects to net8.0.
• Benchmarks work in CLI but fail via MCP
  • Typically a path issue when launching from bin/….
  • Walk up 5 levels from AppContext.BaseDirectory to repo root or auto-detect the root (look for .sln/src).
• dotnet not found in VS
  • Add C:\\Program Files\\dotnet\\ to user PATH and restart Visual Studio.

---

🗺 Roadmap

• upload_project tool: zip → sandbox → run user benchmarks
• DisassemblyDiagnoser tool for hot-path asm dumps
• Publish as dotnet new template or a NuGet tool
• CI hook: post compare_runs deltas on PRs

---

📎 Credits

Built by Aliaksandr Marozka — amarozka.dev

📄 License

MIT