Getting Started

Quick Start

Write your first eval and see results in under 5 minutes.

Create an Eval Config

An eval config is a YAML file with three sections: servers (which MCP endpoints to test), agents (which LLMs to use), and scenarios (what tasks to run and what to assert).

eval.yaml

servers:
  - id: my-server
    transport: http
    url: http://localhost:3000/mcp

agents:
  - id: claude
    provider: anthropic
    model: claude-haiku-4-5-20251001
    temperature: 0

scenarios:
  - id: basic-test
    servers: [my-server]
    prompt: Use the tools to complete this task...
    eval:
      tool_constraints:
        required_tools: [my_tool]
      response_assertions:
        - type: regex
          pattern: success|completed

Run It

Point MCPLab at your config and it will run all scenarios and generate a report.

terminal

npx @inspectr/mcplab run -c eval.yaml

Next Steps

Follow the CLI track to learn about all run options, configuration patterns, and CI integration. Follow the App track to run evaluations from a UI with AI assistants built in.