Skip to content

Conversation

ineersa
Copy link

@ineersa ineersa commented Oct 6, 2025

Implemented application tests as proof of concept instead of inspector tests

Motivation and Context

Currently pipelines are failing
image
I believe that PHP version of tests fits better to PHP SDK and has a lot of benefits.

How Has This Been Tested?

Breaking Changes

No

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

I couldn't test initialize response via Inspector tests, so as discussed in slack tried to implement Application tests as proof of concept.

Category Pros Cons
Inspector Tests - Automatically validate supported methods and error handling - Run as global npx package, no version pin
- Require npm/npx, adds infrastructure dependency
- Generally slow
- Limited method coverage (tools/list, tools/call, resources/*, prompts/*, logging/setLevel)
- Cannot access initialize response, can’t test server capabilities
- Coding agents can’t run npx inside sandboxed envs
Application Tests - Run directly against MCP server responses
- No npm/npx dependency
- Very fast (≈200 ms for ManualStdioExampleTest)
- Easy to implement
- Flexible — can extend with custom client logic (e.g. initialization, logger setup)
- Must implement client-side logic manually once
- Missing handlers won’t automatically raise MCP errors

If we will add some client logic to ApplicationTestCase like initialization, logger setting etc. it will work same as Inspector but without overheads of npm.

Test runs are very fast:
image

Testing is quite easy to implement, server responses indexed by ids and very easy to get for comparison.

@chr-hertel
Copy link
Member

Oh, the scope changed a bit from additional to replacing - can tell you right away that this is not on the table atm.

But I can share my motivation for using the Inspector

  • have a high-level functional test suite
  • use what I actually use to test the servers
  • not re-implement the client side
  • rely on something more neutral instead of the PHP SDK itself - external validation

If you have trouble running the inspector tests locally, I think it is fine to skip them and rely on the pipeline. that's actually also why we split them into separate pipeline. currently those tests take 6-8 seconds locally for me, so not a big deal. not blazing fast tho, true.

Regarding your pro/cons:

  • Inspector Pro: you're missing the part i listed above & that it is ready - we can use it out of the box
    • you're missing the fact that we validate our implementation with an external tool - it's basically contract testing -
  • Inspector Con:
    • well, the first one get's addressed with [Server] Avoid broken inspector cli version by pinning #98
    • I don't see the issue with the dependency, but i'm using npx regularly
    • I give you the limited method coverage, yes
    • Testing the initialize was not the purpose here - it is only one part of the pyramid, the goal is not to test all concerns and skip unit tests
  • Application Tests Pro
    • "Easy to implement" not from the LLOC compared to the other one
    • I feel like your inverting a few inspector cons here :D
  • Application Tests Con
    • I wonder if we can actually use a client implementation later when we have it - feels a bit like incest still

This only to share my thoughts and reasoning, but let's go back a few steps.

Which limitation did you face that you needed to act on this? or was is mostly the npx and inspector bug that threw you off?

@ineersa
Copy link
Author

ineersa commented Oct 7, 2025

This only to share my thoughts and reasoning, but let's go back a few steps.
Which limitation did you face that you needed to act on this? or was is mostly the npx and inspector bug that threw you off?

I was checking how to test MCP server I was working on and was looking at inspector tests, but went with separate implementation which worked really well for my case.

For tests here, yes it was mostly that npx inspector bug which threw me off, so I decided to make this concept into PR.

Also I agree that without client implementation this will be not complete tests.

Maybe we could get back to this when we will have reference MCP Client implementation, then inspector replacement will be super easy, we can close it for now.

Thank you for your feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants