Mastra 推出 Vitest 集成，可在 CI 中运行 Agent 评测

You can now run Mastra evals as part of your CI workflow using Vitest. Each eval behaves like a regular unit test. You can test your agents, workflows, and scorers.

The reporter prints gate and score results at the end of each run so you can see which tests passed or failed, and why.

Catching regressions before you deploy can prevent eg prompt tweaks from changing how your agent behaves in production. Running unit tests as part of your CI means every pull request is validated against the assertions defined in the tests, ensuring only fully functioning code gets deployed.

The alternate path is to use runEvals inside a test and assert using gates and verdicts. But it can be nice to use a familiar testing framework so we shipped this.

Testing LLM responses isn't like testing deterministic code. The same input will often produce a different output, causing exact assertions to fail. The Vitest integration lets you test using tolerances and thresholds, so the tests absorb the variance.

Install @mastra/evals and Vitest (v3 or v4):

Register the MastraEvalsReporter in vitest.config.ts:

Agents

The tests below run a weather agent with gates for tool calls and tool errors, and a scorer for answer relevancy. runEvals returns the result directly; expectEvals wraps the same call as a Vitest assertion and prints a per-gate and per-scorer breakdown in the test output. Each test sets { timeout: 60_000 } to override Vitest's default 5s, which is too short for LLM-backed evals.

Agents with runEvals

Use runEvals when you want a simplified test output:

Output:

Agents with expectEvals

Use expectEvals when you want a per-gate and per-scorer breakdown:

Output:

Workflows

Both APIs accept a workflow as target. The trajectory records workflow_step entries in execution order, with any tool invocations nested as tool_call children. createTrajectoryAccuracyScorerCode compares the actual trajectory against an expectedTrajectory and scores 1.0 when every step (and every named child tool call) matches in order. Used as a gate, it fails the test on any deviation.

Workflows with runEvals

Use runEvals when you want a simplified test output:

Output:

Workflows with expectEvals

Use expectEvals when you want a per-gate breakdown:

Output:

Scorers

Scorers aren't a target, but a custom scorer can be tested on its own with a scorer's .run() method and an expected output. This scorer checks the answer mentions a temperature:

Scorers with .run()

Use .run() with createAgentTestRun and createTestMessage to test the scorer against an expected output:

Output:

Matrix testing

Combine Vitest's test.for with expectEval (or runEvals with a single-item data) for matrix testing against an agent or workflow, running one test per data item so a single regression shows up as one failing test. Both APIs also accept a data array for a single rolled-up pass rate across scenarios.

For more information and full configuration options, see:

Vitest integration; Matrix testing with expectEval; Rate limits and concurrency; runEvals reference; Trajectory accuracy scorers; Evals overview; Gates and verdicts; Quick checks; Custom scorers; Scorer utils; createAgentTestRun; createTestMessage