Implement E2E Evaluation Framework using LLM Frameworks

[akolotov]

The Blockscout MCP server currently lacks end-to-end (E2E) testing that involves a real Large Language Model (LLM). While unit and integration tests are crucial for verifying individual components and live API connections, they do not validate the effectiveness of an LLM in the following areas:

• Tool Description Quality: Assessing whether tool names, descriptions, and parameter schemas are clear and informative enough for LLMs to select and utilize them appropriately.
• Response Format Consistency: Ensuring that tools return data can be interpreted and used effectively by LLMs.
• Tool Selection Accuracy: Evaluating the LLM’s ability to choose the most appropriate tool for a given task, especially when multiple tools are available.
• Multi-Step Workflow Execution: Testing the LLM’s capability to orchestrate sequences of tool calls, handling dependencies and maintaining context across steps.

This testing gap means full confidence cannot be achieved that tool schemas, descriptions, and response formats are optimized for real-world AI agent interactions.

Proposed Solution

A lightweight E2E evaluation framework is proposed, designed to work with various LLM frameworks to simulate real user interactions and assess the MCP server's performance in a realistic environment.

This framework would be housed in a new mcp-evals/ directory and consist of two main components:

1. YAML-based Evaluation Cases: A collection of simple .yaml files, each defining a distinct test case. This format is easy to read and write.

   • Example (mcp-evals/cases/get_latest_block.yaml):

     prompt: "What is the latest block number on Ethereum mainnet?"
     assertions:
       - tool_called: "get_latest_block"
       - params_contain:
           chain_id: "1"
       - response_contains: "The latest block number"

2. A Python Runner Script (mcp-evals/run_evals.py): An executable script that automates the entire evaluation process:

   • Starts the blockscout-mcp-server in the background.
   • Discovers and parses all *.yaml evaluation cases.
   • For each case, it executes the prompt by interacting with the chosen LLM framework (e.g., calling the Gemini CLI as a subprocess, or using a Python library like PydanticAI).
   • It captures the stdout from the CLI, which contains structured information about tool calls.
   • It runs the assertions defined in the YAML file against the captured output.
   • It reports PASS/FAIL for each case and prints a final summary.

Why Gemini CLI (and other LLM frameworks) are Suitable for This Purpose

The Gemini CLI is one suitable approach for this E2E evaluation framework due to its direct interface with the Gemini LLM, scriptability, and structured output. However, it's important to note that other LLM frameworks can also be used for this purpose. For example, PydanticAI (https://ai.pydantic.dev) is a Python library that supports MCP and offers comprehensive documentation, providing an alternative to Node.js-based solutions. The choice of framework will depend on specific project needs and preferences.

• Authentic E2E Testing: Frameworks like Gemini CLI and PydanticAI provide a direct, real-world interface to LLMs, allowing testing of how the actual models interpret prompts and interact with the tools.
• Scriptable and Automatable: As command-line tools or Python libraries, these frameworks can be easily controlled by a Python runner script to automate sending prompts and capturing output.
• Verifiable, Structured Output: The output from these frameworks for tool calls is structured, making it easy and reliable for the runner script to parse and validate against the assertions.

Open Questions

1. Assertion Granularity: How complex should our initial set of assertions be? Is checking for tool_called and basic params_contain sufficient to start, or do we need more powerful assertions (e.g., regex matching, checking the order of tool calls) from the beginning?
2. Strategy for Non-Determinism: LLM text responses can be non-deterministic. To create reliable, non-flaky tests, we will adopt a two-pronged assertion strategy:
   • For data-retrieval prompts: We will craft prompts that explicitly ask the LLM to return a structured JSON object. For example, instead of asking "What is the balance?", we will ask "Return the balance in the format: {"balance": "..."}". This makes the final output predictable and easy to assert against.
   • For summarization prompts: For more open-ended prompts (e.g., "Summarize this address's activity"), we will use more flexible assertions, such as checking for the presence of specific keywords or phrases in the response, rather than asserting on the exact text.
3. CI Integration Strategy: Given that these tests will be slower and may require API keys, what is the best way to integrate them into our CI/CD pipeline? Should they run on every commit, only on a schedule (e.g., nightly), or be triggered manually before a release?