MCPLab

Reference

Configuration Schema

Field reference for evaluation YAML and workspace evaluation settings.

servers[ ]

  • id (string, required) — unique identifier used in scenario server lists.
  • transport (string, required) — connection type. Use "http" for HTTP/SSE.
  • url (string, required) — full URL of the MCP endpoint.
  • token (string, optional) — bearer token for authentication. Use a literal string or $ENV_VAR to read from the environment.
  • inspectr.enabled (boolean, optional) — route this server's MCP traffic through a local Inspectr proxy for runs queued from the App. Defaults to false. See Inspectr Request Inspection.

agents[ ]

  • id (string, required) — unique identifier referenced by scenarios.
  • LLM agent: omit type or use type: llm; requires provider and model. Supported API providers are anthropic, openai, azure_openai, and azure_foundry.
  • Azure Foundry: model is the required deployment name. Each agent can target a different deployment; backend endpoint and API-key settings are shared. See Azure Foundry agent setup.
  • Browser agent type: type: browser (required). Browser agents are executed through Rover.
  • Browser agent provider (string, required): built-in or learned Rover provider identifier.
  • Browser agent url (string, required): page Rover opens when the agent needs attention.
  • new_conversation_between_scenarios (boolean, browser only, optional): defaults to starting a new conversation and can be overridden when a run starts.
  • temperature (number, LLM only, optional): sampling temperature. Defaults to 0.
LLM and browser agents
agents:
  - id: api-agent
    provider: anthropic
    model: claude-haiku-5-5

  - id: foundry-agent
    provider: azure_foundry
    model: my-chat-deployment

  - id: foundry-agent
    provider: azure_foundry
    model: my-chat-deployment

  - id: browser-agent
    type: browser
    provider: chatgpt-com
    url: https://chatgpt.com
    new_conversation_between_scenarios: true

scenarios[ ]

  • Inline scenario: id, prompt, optional name, optional mcp_servers, optional eval and extract.
  • Referenced scenario: ref with optional mcp_servers override.
  • mcp_servers entries can be { ref: <server-id> } or inline server objects.
  • Legacy top-level servers pool is deprecated; prefer scenario-owned mcp_servers.

eval Assertions

  • tool_constraints.required_tools (string[], optional) — tool names the agent must call. An entry such as search | lookup accepts either alternative.
  • tool_constraints.forbidden_tools (string[], optional) — tool names the agent must not call.
  • tool_sequence (string[], optional) — ordered steps that must appear in the run in that order, with other tools allowed in between. A step such as search | lookup accepts either alternative.
  • tool_input_assertions and tool_output_assertions (arrays, optional): each entry names one tool and has type contains with value, regex with pattern, or jsonpath with path and optional primitive equals (string, number, or boolean). Output JSONPath addresses the full MCP response, for example $.structuredContent.total.
  • response_assertions supports: contains, not_contains, starts_with, ends_with, equals, regex, jsonpath, jsonpath_exists, jsonpath_not_exists.
  • For detailed examples of each assertion, see Tool and Response Assertions.

Workspace Judge and Decision Settings

Store evaluation_judge_agent_name and optional evaluation_decision in the library directory's .mcplab-app-settings.yaml, normally mcplab/.mcplab-app-settings.yaml. These settings apply across evaluation entry points; they are separate from eval.yaml and a scenario's eval block.

evaluation_decision contains provider, model, optional enabled, optional Ollama baseUrl, passThreshold (default 0.9), and failThreshold (default 0.1). Providers are openai, cloudflare, typesafe, ollama, and azure_foundry. For Azure Foundry, model is the decision deployment name; its endpoint comes from backend environment variables.

An enabled engine requires an existing LLM judge. Thresholds must satisfy 0 <= failThreshold < passThreshold <= 1. Set enabled: false to retain settings while using the judge directly; omission means enabled. Remove the decision setting to clear it. Keep credentials in environment variables. See workspace YAML examples.

Reference Syntax

Use ref inside agents and scenarios list items to pull from library ids.

referenced scenario with mcp override
agents:
  - ref: claude-sonnet-55

scenarios:
  - ref: add-calculations
    mcp_servers:
      - ref: kpi-api-prod