Add support for a stdio testing

[remyleone]

Is your feature request related to a problem? Please describe.

It would be helpful when I'm running a MCP server in stdio mode to have the option to check that the conformance tests are passing.

Describe the solution you'd like

npx @modelcontextprotocol/conformance server --stdio ./my_server

Describe alternatives you've considered

I've tried the run a server in a SSE fashion using the examples of official go-sdk but it didn't worked:

• https://github.com/modelcontextprotocol/go-sdk/tree/main/examples/server/sse

Additional context

It would be helpful to have consistant and conformance friendly examples in all SDKs. Could you ensure that the examples in the official documentation are passing the conformance test and show casing report for them?

[yagna-1]

+1 on this — stdio is the dominant transport for Python and Go MCP servers in the wild, and right now the only path to running the conformance suite against them is to wrap the stdio server in an HTTP shim, which most maintainers won't do just for testing.

A few notes from a Python-side perspective that might help shape the design, since I've been working in this area:

1. The CLI surface in #258 is the right one. --stdio "<command>" (matching the existing --command for client testing) keeps the runner ergonomically symmetric:

npx @modelcontextprotocol/conformance server --stdio "uv run my-server"  --suite active
npx @modelcontextprotocol/conformance server --url    http://localhost:3000/mcp --suite active

The two flags are mutually exclusive. The runner spawns the subprocess, frames JSON-RPC over its stdin/stdout, and otherwise reuses the same scenario set.

2. Watch out for the SSE/Streamable-HTTP-only checks. A handful of current scenarios assume HTTP semantics (status codes, headers, multi-session). Those would need a transports: ['http'] filter, or the suite would emit spurious failures for stdio servers that are otherwise spec-conformant. Probably worth tagging at the scenario level rather than gating in the runner.

3. Stderr is signal. Python servers commonly log diagnostics to stderr. Capturing and surfacing that in stderr.txt (the way client testing already does) is much more useful than discarding it — when a stdio scenario fails, the server's stderr is usually the first thing a maintainer wants to see.

4. Re: the broader ask in the issue (consistent conformance across SDK examples) — separately, I've published a community pytest plugin (pytest-mcp-plugin on PyPI) that drives MCP servers over stdio/HTTP from inside the Python test runner. It's not a substitute for this conformance suite — it's the layer below, useful for SDK-author-style tests and per-server CI rather than spec-coverage. Happy to write a "Python integration" section for SDK_INTEGRATION.md once stdio support lands here, showing how the two compose (conformance for the protocol; pytest plugin for per-tool/per-fixture assertions). Repo: https://github.com/yagna-1/mcp-test

Glad to draft a PR for the stdio runner if @pcarleton / @felixweinberger want to scope it together first per the contributor guide.

[yagna-1]

Update on the SDK_INTEGRATION.md Python section I offered above:

I've gone ahead and built end-to-end against the suite in the meantime, with pytest-mcp-plugin v0.2.x running npx @modelcontextprotocol/conformance server --suite active --expected-failures conformance-baseline.yml against a bundled FastMCP demo server in CI. The expected-failures workflow worked exactly as documented (passes when only baseline scenarios fail; fails on regressions and stale entries) — so the Python integration story end-to-end is now real and reproducible, not theoretical.

If it's useful to maintainers, I can put up a small docs PR that adds a "Python (pytest)" subsection to SDK_INTEGRATION.md, mirroring the existing TypeScript guidance. Roughly:

1. A "Server Testing — Python" snippet showing the GitHub Action wired up against a Python FastMCP server (boot + curl-wait + modelcontextprotocol/conformance@v0.1.11 action).
2. A short note on the baseline-file workflow as it actually plays out for a Python SDK author (matches the language-agnostic section already in the doc, just pointing to a Python example repo).
3. A pointer to pytest-mcp-plugin as the layer-below pytest harness for SDK authors who want per-tool/per-fixture assertions in addition to spec coverage.

Happy to scope before opening the PR if @pcarleton / @felixweinberger want to weigh in on what should and shouldn't live in that doc — it's clearly meant to stay SDK-agnostic, so the Python pointer would be a brief paragraph rather than a full guide. Just want to confirm a Python subsection is in scope before sending the PR.

Either way, the stdio runner request in the original issue body would unlock a much larger fraction of real-world Python servers — happy to help with that PR too once a maintainer scopes it.