Reference
Configuration Schema
Field reference for evaluation YAML and workspace evaluation settings.
servers[ ]
- id (string, required) — unique identifier used in scenario server lists.
- transport (string, required) — connection type. Use "http" for HTTP/SSE.
- url (string, required) — full URL of the MCP endpoint.
- token (string, optional) — bearer token for authentication. Use a literal string or $ENV_VAR to read from the environment.
- inspectr.enabled (boolean, optional) — route this server's MCP traffic through a local Inspectr proxy for runs queued from the App. Defaults to false. See Inspectr Request Inspection.
agents[ ]
- id (string, required) — unique identifier referenced by scenarios.
- LLM agent: omit type or use
type: llm; requires provider and model. Supported API providers areanthropic,openai,azure_openai, andazure_foundry. - Azure Foundry:
modelis the required deployment name. Each agent can target a different deployment; backend endpoint and API-key settings are shared. See Azure Foundry agent setup. - Browser agent type:
type: browser(required). Browser agents are executed through Rover. - Browser agent provider (string, required): built-in or learned Rover provider identifier.
- Browser agent url (string, required): page Rover opens when the agent needs attention.
- new_conversation_between_scenarios (boolean, browser only, optional): defaults to starting a new conversation and can be overridden when a run starts.
- temperature (number, LLM only, optional): sampling temperature. Defaults to 0.
agents:
- id: api-agent
provider: anthropic
model: claude-haiku-5-5
- id: foundry-agent
provider: azure_foundry
model: my-chat-deployment
- id: foundry-agent
provider: azure_foundry
model: my-chat-deployment
- id: browser-agent
type: browser
provider: chatgpt-com
url: https://chatgpt.com
new_conversation_between_scenarios: truescenarios[ ]
- Inline scenario:
id,prompt, optionalname, optionalmcp_servers, optionalevalandextract. - Referenced scenario:
refwith optionalmcp_serversoverride. mcp_serversentries can be{ ref: <server-id> }or inline server objects.- Legacy top-level
serverspool is deprecated; prefer scenario-ownedmcp_servers.
eval Assertions
- tool_constraints.required_tools (string[], optional) — tool names the agent must call. An entry such as
search | lookupaccepts either alternative. - tool_constraints.forbidden_tools (string[], optional) — tool names the agent must not call.
- tool_sequence (string[], optional) — ordered steps that must appear in the run in that order, with other tools allowed in between. A step such as
search | lookupaccepts either alternative. tool_input_assertionsandtool_output_assertions(arrays, optional): each entry names onetooland has typecontainswithvalue,regexwithpattern, orjsonpathwithpathand optional primitiveequals(string, number, or boolean). Output JSONPath addresses the full MCP response, for example$.structuredContent.total.- response_assertions supports: contains, not_contains, starts_with, ends_with, equals, regex, jsonpath, jsonpath_exists, jsonpath_not_exists.
- For detailed examples of each assertion, see Tool and Response Assertions.
Workspace Judge and Decision Settings
Store evaluation_judge_agent_name and optional evaluation_decision in the library directory's .mcplab-app-settings.yaml, normally mcplab/.mcplab-app-settings.yaml. These settings apply across evaluation entry points; they are separate from eval.yaml and a scenario's eval block.
evaluation_decision contains provider, model, optional enabled, optional Ollama baseUrl, passThreshold (default 0.9), and failThreshold (default 0.1). Providers are openai, cloudflare, typesafe, ollama, and azure_foundry. For Azure Foundry, model is the decision deployment name; its endpoint comes from backend environment variables.
An enabled engine requires an existing LLM judge. Thresholds must satisfy 0 <= failThreshold < passThreshold <= 1. Set enabled: false to retain settings while using the judge directly; omission means enabled. Remove the decision setting to clear it. Keep credentials in environment variables. See workspace YAML examples.
Reference Syntax
Use ref inside agents and scenarios list items to pull from library ids.
agents:
- ref: claude-sonnet-55
scenarios:
- ref: add-calculations
mcp_servers:
- ref: kpi-api-prod