agentevals CLI
agentevals scores AI agent behavior from OpenTelemetry traces and ships MCP tools plus the /eval and /inspect skills.
- 168 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The server reads trace files and agent sessions that may contain request and response text.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/agentevalsDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The server reads trace files and agent sessions that may contain request and response text.
This is third-party code. Review the repository files before installing.
What it does
agentevals takes the OpenTelemetry traces an agent already emits and compares its actions with a golden set of checks: tool-call order, final answer and quality rubrics. Scoring reads recorded data, so the agent is not rerun and no tokens are spent. It ships a CLI for CI, a web UI for session inspection and the `agentevals mcp` server with `list_metrics`, `evaluate_traces`, `list_sessions`, `summarize_session` and `evaluate_sessions` tools. The `/eval` and `/inspect` skills in `.claude/skills/` explain the results inside Claude Code.
Who it is for. Teams building agents on Google ADK, LangChain or the OpenAI Agents SDK that want to catch behavior regressions before release.
Good fit when
- An agent stopped calling the right tool and the cause is unclear.
- CI has to stop a release when the call trajectory diverges from the golden one.
- Two agent sessions are compared for regression against a reference session.
- Evaluation results are needed in an existing telemetry pipeline as `gen_ai.evaluation.result` events.
Not a fit when
- The agent does not emit OpenTelemetry traces and you do not plan to add them.
- You need a hosted platform with dashboards: agentevals runs locally with an embedded UI.
- Checking a single model answer without tracing is enough.
Example request
Score the samples/helm.json trace against the golden set and explain why the call trajectory failed.Limitations
The project is under active development and the README warns of possible breaking changes. Traces must follow the OpenTelemetry GenAI semantic conventions; otherwise turns, tool calls and tokens are not recognized. Rubric and judge-model scoring needs LLM access. The MCP server is part of the `[live]` extra, and without it the assistant has no tools. Running from a clone uses `uv`. Traces may contain request and response text, so keep private sessions on your own machine.
How to disable. Remove the `agentevals` entry from `.mcp.json` and the `/eval` and `/inspect` skills, then uninstall the package with `pip uninstall agentevals-cli`.
FAQ
Does the agent have to be rerun for scoring?
No. Scoring reads already recorded traces, so it costs no agent tokens and can be repeated on the same data. `agentevals serve` accepts live sessions.
How do I run the check in CI?
`agentevals run trace.json --eval-set eval_set.json -m tool_trajectory_avg_score` returns an exit code based on the threshold. The `--output json` option gives machine-readable results.
How do I connect it to Claude Code?
The `.mcp.json` file at the repository root launches `agentevals mcp`. The `/eval` and `/inspect` skills live in `.claude/skills/` and build on its tools.
Can I write my own metric?
Yes. Any program that reads JSON from stdin and writes a score to stdout plugs in as `type: code`. `agentevals evaluator init` scaffolds one.
Does scoring work without a server?
Yes. `agentevals run` works offline on files. The server is for the web UI, live OTLP sessions and the MCP tools that read sessions.
Discussion
No questions or reviews yet.
Related
CLI toolsVercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
A local debugger for TypeScript agents: run step trees, CI trajectory checks and portable evidence bundles, with no account.
A skill that teaches Claude Code and Cursor to wire in Tracely tracing and turn failing agent runs into CI tests
An MCP server that checks whether the agent is really done: drives the real app and returns pass, fail or couldn't tell with the file and line
CLI, MCP server and typed Node.js API so an agent can verify its changes in a live mobile app on iOS, Android and more
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| agentevals (this card) | no | not required | 168 | |
| agent-eval | yes | 277 | a different agent-evaluation project; agentevals scores from OpenTelemetry traces and exposes MCP tools | |
| Langfuse Skills | yes | 333 | skills for the Langfuse observability platform; agentevals is local and accepts standard OTLP |
README in short
agentevals acts as an OTLP destination: traces and logs go in, and results come back as OpenTelemetry events. The serverless mode suits CI.