CLI

CI/CD

Run MCPLab in GitHub Actions and other CI pipelines.

GitHub Actions

The example below runs evaluations on every push. It uploads the run directory as an artifact so you can inspect reports from the Actions UI.

.github/workflows/eval.yml

name: MCPLab Evaluation

on: [push, pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Run evaluations
        run: npx @inspectr/mcplab run -c eval.yaml --run-note "ci-${{ github.sha }}"
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Upload run artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: mcplab-run
          path: runs/

Exit Codes and Failure Detection

mcplab run exits 0 when all scenarios pass. Any failing scenario produces a non-zero exit code, which causes the CI step to fail.

Use if: always() on the artifact upload step so reports are preserved even when the eval step fails.

Environment Variables in CI

Store API keys as repository secrets and pass them as environment variables. For server bearer tokens defined with $TOKEN_VAR in your config, add the matching secret.

env block in workflow step

env:
  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  SERVER_API_TOKEN: ${{ secrets.SERVER_API_TOKEN }}