agent-eval CLI
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
- 277 ★ on the repository
- Last commit
Extended access
Why this level
- Agents run in a sandbox and execute task commands and code
- Provider keys and the Vercel token live in .env and reach the sandbox
- Every run spends paid model tokens
Install
- Works in
Manual install
npx @vercel/agent-eval init my-agent-evalsScaffolds a project with examples. Then `npm install`, a `.env` with keys and `npx @vercel/agent-eval --dry` to check the plan.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Agents run in a sandbox and execute task commands and code
- Provider keys and the Vercel token live in .env and reach the sandbox
- Every run spends paid model tokens
This is third-party code. Review the repository files before installing.
What it does
agent-eval is a Vercel CLI that tests how well coding agents handle tasks on your framework or library. You describe a task in `PROMPT.md`, write checks in `EVAL.ts` with Vitest, and `npx @vercel/agent-eval` runs the agent in a sandbox and verifies the result. An experiment is a file in `experiments/`: agent, model, number of runs and isolation, Docker or Vercel Sandbox. Checks see both the files and how the agent worked: which commands it ran, which files it read and how many tool calls it made. For open-ended quality criteria there is an agentic judge that scores the final state or the transcript. Results land in `summary.json`, and the `playground` command opens a web viewer. This suits A/B comparisons: with and without documentation, with and without an MCP server, Sonnet against Opus.
Who it is for. Authors of frameworks and libraries who need to know whether an agent writes working code with their API and whether documentation or an MCP server helps.
Good fit when
- A new API version shipped and you need to know whether agent compatibility broke
- You are comparing whether adding an MCP server raises the share of successful tasks
- You want to find out whether a cheap model manages where you use an expensive one
- CI should run a set of typical tasks, and repeated runs should reuse cached results
- You need an assertion about agent behavior: too many calls, or the wrong scaffold command
Not a fit when
- You need to evaluate skills for quality and overlap: NVIDIA SkillEvaluator does that
- A task cannot be tested: every task needs an `EVAL.ts` with checks
- You want to compare models in plain chat without code: the tool is built for coding tasks in a sandbox
Example request
Create an evals project with `npx @vercel/agent-eval init my-agent-evals` and add a Button component task checked with VitestLimitations
A run needs an agent key (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY` and the like, or `AI_GATEWAY_API_KEY`) and a sandbox: Docker or a `VERCEL_TOKEN`. Every run spends tokens, so start with the `--dry` flag. The classifier that filters out infrastructure and rate-limit failures turns on only with `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN`. OpenCode and fx work only through Vercel AI Gateway.
How to disable. Remove the package with `npm uninstall @vercel/agent-eval` and delete the experiments project folder. Revoke the tokens from `.env` at the provider if they served only for evaluation.
FAQ
How can I check the setup without spending money?
The `--dry` flag shows what would run and makes no API calls. The `--smoke` flag takes the first eval alphabetically and runs it once per model.
Which agents are supported?
claude-code, codex, gemini and cursor work directly. opencode, fx and pi-coding-agent are also available through Vercel AI Gateway, and you can define your own agent.
Can I do without a Vercel account?
Yes. Provide a provider key such as `ANTHROPIC_API_KEY` and set `sandbox: 'docker'` in the experiment config. Then `VERCEL_TOKEN` is not needed.
What happens on a repeated run?
Results with a matching fingerprint are reused and not paid for twice. The `--force` flag ignores the cache and reruns everything.
How does the judge score a result?
The `toSatisfyCriterion` matcher returns pass or fail against a text criterion, and `toScoreAtLeast` compares a 0 to 1 score with a threshold. You pick the subject: `environment` for the final state or `transcript` for how the agent worked.
Discussion
No questions or reviews yet.
Related
CLI toolsagentevals scores AI agent behavior from OpenTelemetry traces and ships MCP tools plus the /eval and /inspect skills.
A local debugger for TypeScript agents: run step trees, CI trajectory checks and portable evidence bundles, with no account.
Reviews the git diff and the conversation with the agent: catches bugs and gaps between your request and the agent's result. CLI, skill, GitHub Action
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
QA agents, skills and an MCP server: test generation, coverage gaps, flaky tests, security scans and chaos testing
33 engineering-discipline skills for Claude Code, Codex and Gemini CLI plus vibe-cli for CI: review, tests, coverage, deploy, rollback
README in short
A detailed English README: quick start, CLI flags, task structure and example checks, an experiment configuration reference, A/B comparisons, result formats, environment variables and tips. The examples section shows how to assert on agent behavior and how to add a judge.