LangSmith Integration

Why Pair MCPLab with LangSmith?

MCPLab answers whether an MCP-powered agent succeeded. LangSmith helps you understand how it got there. Together, you can keep deterministic evaluation results locally while exploring the full model and tool-call journey in a trace viewer.

This is especially useful when a scenario fails intermittently, the agent selects the wrong tool, a tool returns unexpected data, or two models achieve the same pass rate through different trajectories.

• Keep local trace.jsonl and results.json as the authoritative evaluation artifacts.
• Inspect prompts, assistant responses, MCP tool calls, tool inputs, tool results, and final answers in one trace.
• Compare providers and models using the same scenarios and trace structure.
• Use LangSmith’s filtering, token, latency, and cost views to investigate regressions.

Configure the Integration

LangSmith export is opt-in. MCPLab exports traces only when LANGSMITH_TRACING is true and LANGSMITH_API_KEY is set. Put these values in your local .env file or shell environment, and never commit the API key.

• LANGSMITH_ENDPOINT is optional; use it for a regional or self-hosted LangSmith endpoint.
• LANGSMITH_PROJECT selects the destination LangSmith project. MCPLab does not create a project per evaluation.
• LANGSMITH_WORKSPACE_ID is optional and can identify the target workspace when your account requires it.

LANGSMITH_TRACING=true
LANGSMITH_API_KEY=lsv2_...
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_PROJECT="my-mcplab-evaluations"

# Optional for workspace-specific deployments
LANGSMITH_WORKSPACE_ID=your-workspace-id

Run an Evaluation

Run MCPLab normally from the CLI or App. The exporter is asynchronous and best-effort, so a LangSmith outage or export error does not change the evaluation result or prevent local artifacts from being written.

npx @inspectr/mcplab run -c evals/eval.yaml

Start mcplab app, open Run Evaluation, and launch a run as usual.

What MCPLab Sends

Each executable scenario run is exported as a parent chain trace with nested LLM and MCP tool spans. LLM spans use LangChain-style messages so tool calls, tool results, and multimodal content can be rendered as a conversation.

• Scenario chain: scenario ID, run ID, request ID, agent, provider, model, configuration hash, git commit, CLI version, pass/fail, final answer, and metrics.
• LLM spans: accumulated messages, assistant text, tool calls, usage metadata, provider, and model name.
• MCP tool spans: tool name, server identity in metadata, direct tool arguments, direct MCP results, status, and duration.
• Errors close their spans and are reported as trace errors without changing MCPLab’s local pass/fail semantics.

Open a Trace from MCPLab

After a run is exported, MCPLab adds a LangSmith trace link to the run actions menu on the Results page and to the Result Detail page. Select LangSmith trace to open the corresponding trace in a new tab.

• The link appears only when LangSmith returned a trace URL for that run.
• Trace URLs are associated with the scenario request ID, so multi-scenario evaluations can link each scenario run separately.
• Traces created before export was enabled cannot be linked retroactively.

Privacy and Data Handling

MCPLab exports full prompts, model messages, MCP tool inputs, and MCP tool results when tracing is enabled. Review your data-handling requirements before enabling it for sensitive evaluations. Disable the integration by setting LANGSMITH_TRACING=false or removing LANGSMITH_API_KEY.