Coder Eval CLI
Runs Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
- 160 ★ on the repository
- Last commit
- Apache-2.0
Broad access
Why this level
- The framework launches real agents that execute code and commands in a sandbox, and with a misconfigured sandbox that is your own machine
- Runs spend paid-agent tokens and use their credentials
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/coder-evalDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The framework launches real agents that execute code and commands in a sandbox, and with a misconfigured sandbox that is your own machine
- Runs spend paid-agent tokens and use their credentials
This is third-party code. Review the repository files before installing.
What it does
Coder Eval is a Python framework for evaluating coding agents and their skills. A task is described in YAML: prompt, files, criteria. The `coder-eval run` command launches a real agent in a sandbox, then scores the files and commands it produced against weighted criteria from 0 to 1. Claude Code, OpenAI Codex, Google Antigravity (Gemini), OpenCode and Pi are supported, and switching agent comes down to the `agent.type` field. The `coder-eval plan` command validates a task without spending tokens. The `coder-eval report` command shows the outcome. There is a `skill_triggered` criterion that answers whether a skill fired in the agent, an A/B experiment layer (model vs model, prompt vs prompt, Claude Code vs Codex), datasets, per-tool telemetry and a dialog mode. The framework runs in Docker isolation and connects to CI through the `coder_eval` GitHub Marketplace action and a packaged Claude Code plugin. The author calls it "Playwright for coding agents" and stresses it has no fixed leaderboard: you bring the tasks and the scoring.
Who it is for. Skill and CLI authors, teams choosing between agents on their own tasks, and anyone who wants CI to block a merge when agent quality drops.
Good fit when
- You ship a skill and want to know whether it triggers in Claude Code, Codex and OpenCode
- You need to compare two models or two prompts on your tasks with repeats and a report
- A team needs a CI gate that blocks a merge when the agent fails a task set
- A benchmark must use your data and your criteria, and a fixed set like SWE-bench will not do
Not a fit when
- You need a general model leaderboard: the framework has no shared ranking, only your tasks
- A one-off manual test of a single prompt is enough, with no sandbox or reports
- You need Python below 3.13: the package requires Python 3.13 or newer
Example request
Check whether my skill triggers in Claude Code and Codex: write a coder-eval task with a skill_triggered criterion and run both agentsLimitations
Each run launches a real agent, so it spends tokens and time and needs the agent own credentials. Claude Code is installed as a separate CLI, as is OpenCode, while Codex and Antigravity have extra dependency groups at install. You write the scoring criteria, and weak criteria give a weak picture.
The Docker sandbox requires Docker and permission to run containers. Run results depend on the model and chance, so conclusions need repeats. The framework scores files and commands the agent produced and does not score the quality of the whole conversation, though a dialog mode exists.
The project is under active development: changes by version are listed in `CHANGELOG.md`.
How to disable. Remove the package with `uv tool uninstall coder-eval` or `pip uninstall coder-eval`. The `runs/` directory with run results and your YAML tasks stay in the project; delete them by hand.
FAQ
How do I install Coder Eval?
Run `uv tool install coder-eval` or `pip install coder-eval`. For Codex and Antigravity add extras, for example `uv tool install "coder-eval[codex,antigravity]"`.
Which agents are supported?
Claude Code, OpenAI Codex, Google Antigravity (Gemini), OpenCode and Pi. The agent type changes with one `agent.type` field while tasks and criteria stay the same.
Can I validate a task without spending tokens?
Yes, `coder-eval plan <file>` validates the task description without launching an agent.
How is this different from SWE-bench?
SWE-bench and SkillsBench rank models on a shared task set. Coder Eval takes your tasks and your criteria and ranks nothing in advance.
How do I connect it to CI?
Through the `coder_eval` GitHub Marketplace action or the packaged Claude Code plugin. See `docs/CI_GATE.md` and `docs/PLUGIN.md`.
Discussion
No questions or reviews yet.
Related
CLI toolsVercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
Behavioral eval lab Quorum for the superpowers project: drives real agent CLIs and grades them against private criteria
A Harbor fork for evaluating coding agents in sandboxes: Claude Code, Codex and others run inside tasks with no internet.
A Python CLI that measures what a skill changes: runs a case with and without it, lints answer leaks in prompts and diffs reports
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| Coder Eval (this card) | no | 160 | ||
| agent-eval | yes | 280 | also runs agents on tasks without multi-agent support or an A/B layer | |
| EvalView | no | not required | 142 | focuses more on agent behavior regression tests |
README in short
The UiPath framework runs different coding agents from one task description, scores the files they create against weighted criteria and can gate CI. The author lists scenarios: a benchmark on your own data, a skill trigger check, A/B comparison of agents and models, and a quality gate in the pipeline.