OpenEval CLI
OpenEval runs coding agents on your tasks with scoring rubrics and records cost, tokens and tool reliability
- 79 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Runs agents in Docker containers that execute code and commands from the tasks
- Uses your connected models and spends tokens on candidates and the judge
Install
- Works in
Manual install
bun add --exact @hona/openevalNeeds Bun 1.4.2+, Docker and connected OpenCode models; then `bunx --bun @hona/openeval image`.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Runs agents in Docker containers that execute code and commands from the tasks
- Uses your connected models and spends tokens on candidates and the judge
This is third-party code. Review the repository files before installing.
What it does
OpenEval is a typed SDK, CLI and evidence viewer for evaluating agents on OpenCode. A benchmark consists of evals, each with a `prompt.md` task for the candidate, a `judge.md` rubric with named criteria for an LLM judge, a `judge.ts` plain function returning scores and JSON and an optional `eval.ts` for workspace preparation and early stopping. A code judge receives the response, events, message history, tool calls and workspace snapshots. Booleans become 0 or 1, numbers range from 0 to 1 and `null` marks an unresolved criterion. Code-only benchmarks need no judge model. The runner records cost, tokens, tool reliability and timing itself. `benchmark.ts` sets models, the judge and the repetition count. The `bunx --bun @hona/openeval image`, `plan`, `run` and `view` commands build the image, show the plan, run it and open the viewer at `http://127.0.0.1:4173`. A repeated `run` continues the same aggregate and keeps scores already collected. The `.opencode/skills/eval-writing` folder holds a skill for writing evals.
Who it is for. Developers of agents and tools on OpenCode who need to compare models and prompts on their own tasks with reproducible scoring.
Good fit when
- Three models need to be compared on your project tasks with the cost of each visible
- Agent scoring follows a rubric of criteria such as asking for the SQL dialect
- A check is unambiguous and a `judge.ts` function suffices with no judge model
- An interrupted run should continue without losing the scores already collected
Not a fit when
- Agents do not run on OpenCode: the runner and models are tied to it
- The machine has no Docker or Bun: both are required to run
- You need a quick one-off test of a single answer: a manual check is enough
Example request
Create an eval with the rubric "asks for the SQL dialect before the query" and run it on two models with three repetitions eachLimitations
- Bun 1.4.2 or newer, Docker and connected OpenCode models are required.
- Production OpenCode packages are pinned at 2.0.22 and the `image` command must run after an upgrade.
- A judge model is required when an eval contains `judge.md`.
- The viewer opens locally at `127.0.0.1:4173`, and an interrupted runner waits until its heartbeat is two minutes old.
- Judge model scoring costs tokens, and you set the number of repetitions and models in `benchmark.ts`.
How to disable. Remove the package with `bun remove @hona/openeval` and delete the `evals/` folder and `benchmark.ts` from the project; remove the Docker image with `docker rmi`.
FAQ
Which files does one eval need?
`prompt.md` with the task, `judge.md` with a rubric or `judge.ts` with a function and optionally `eval.ts` to prepare the workspace.
Is a judge model required?
Only when an eval has `judge.md`. A benchmark of plain functions needs none.
What does the runner record?
Cost, tokens, tool reliability, timing, source identity and recording links.
How do I view results?
With `bunx --bun @hona/openeval view`: the viewer opens at `http://127.0.0.1:4173` with a radar chart, heatmap and scorecard.
What if a run was interrupted?
Run `run` again: it waits until the previous runner heartbeat is two minutes old, keeps the records and collects the missing work.
Discussion
No questions or reviews yet.
Related
CLI toolsRuns Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
Nine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.
A benchmark that tests browser agents on 281 everyday tasks across live websites and records each run in five layers.
A Python SDK and CLI that evaluate agent self-improvement on ready benchmarks: a candidate goes in, scores come back to your loop.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| OpenEval (this card) | no | 79 | |
| agent-eval | yes | 280 | a Vercel CLI that counts the success rate for Claude Code and Codex, while OpenEval scores by rubrics on OpenCode |
| Evals Skills | no | 1,770 | skills from an evals course with no runner or viewer |
README in short
The README shows the prompt and judge pairing, two kinds of judges, a vocabulary of terms, model setup in `benchmark.ts` and the `image`, `plan`, `run`, `view` commands. It then covers removing a model from an aggregate and rejudging.