Caliper CLI
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
- 228 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Launches real agents on tasks and spends tokens; `assert:` code runs locally
- The spec and results are stored in the project, and anonymous telemetry can be turned off per the docs
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/caliperDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Launches real agents on tasks and spends tokens; `assert:` code runs locally
- The spec and results are stored in the project, and anonymous telemetry can be turned off per the docs
This is third-party code. Review the repository files before installing.
What it does
Caliper checks whether an agent skill works every time, not just once. Next to the skill sits an `.eval.yaml` file that lists the skills to install (yours plus neighbours that might compete for the same prompts) and the tasks to run. Each task has a prompt, an optional `setup:` step and at least one check. `expect:` is graded by an LLM judge, `assert:` runs locally as Python in the attempt's workdir with a 30-second limit, and `activates:` asserts which skills the agent chose to load. The skill is installed where the agent looks for it, never pasted into the prompt, so the real install is what gets tested. `caliper run` repeats each task k times (3 by default) and scores the success rate apart from how often the skill fired. With `--ablate` the run repeats without the skill, and `caliper compare` shows the difference in success, tokens and wall time. The spec never names an engine: you run it on Claude Code, Codex, Pi or Hermes by switching the model at launch. The repository also ships two skills for the agent itself: `grill-skill` interviews you about scenarios, writes the spec and runs it with and without your skill, and `evaluate-skill` reads the results. Run results are saved as JSON under `.caliper/results/`.
Who it is for. Authors of skills for Claude Code and Codex who want proof that a skill fires on time and earns its place in context, and teams whose skills break after a model change.
Good fit when
- A skill worked when you tried it, and you want to know how often it works across ten runs
- Two skills compete for one prompt and you need to check which one the agent picks with an `activates:` assertion
- Before editing a skill's text you want an `--ablate` baseline: if the score stays the same, the skill wastes context
- After a model update you want to confirm the skill still passes its tasks
- You suspect a skill is open to prompt injection and want a dedicated task for it in the spec
Not a fit when
- You need to test application code: Caliper runs an agent with its skills and does not replace unit tests
- The token budget is tight: every task runs k times, plus the judge and the run without the skill
- The skill is still a draft: without stable scenarios the spec goes stale quickly
Example request
Test the inbox-triage skill: write a spec, run it with --ablate and show how much the skill lifts success and how many tokens it savesLimitations
- Every task launches a real agent k times, and `--ablate` doubles that, so a run spends a noticeable number of tokens.
- An `expect:` check relies on an LLM judge and can be wrong; for exact facts the README advises writing `assert:` instead.
- `assert:` code runs on your machine with a 30-second limit, so read someone else's spec before running it.
- Anonymous usage telemetry is described in docs/telemetry.md together with the opt-out.
How to disable. Run `pipx uninstall caliper-eval` to remove the CLI and delete the `.caliper/` folder in the project along with the results. The `grill-skill` and `evaluate-skill` skills are removed from your agent's skills folder.
FAQ
How is Caliper different from promptfoo?
promptfoo compares prompts and models against sets of checks. Caliper installs the skill into the agent as a normal install and checks whether the agent picked it and gained from it, including against a run without the skill.
How do I check that a skill is needed at all?
Run `caliper run file.eval.yaml --ablate skill-name` at k=3, then a normal run, and compare them with `caliper compare`. If success and tokens match, the skill does not earn its place in context.
Which agents does Caliper run on?
Claude Code, Codex, Pi and Hermes. The engine is picked with `--model` at launch; by default the skill runs on Claude Code and the judge on the same backend.
What is an activates check?
It asserts which skills the agent loaded for a prompt. It needs no judge, so it costs much less than a graded task.
What does grill-skill do?
It interviews you about scenarios, writes an `.eval.yaml` spec and runs it with and without your skill. Call `/grill-skill ./inbox-triage/SKILL.md` after installing the skills with `npx skills@latest add edonadei/caliper`.
Discussion
No questions or reviews yet.
Related
CLI toolsA CLI to evaluate and evolve agent skills: declarative YAML cases, runs across engines and structured reports
Validates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
A skill runs a third-party coding-agent harness through 30 benchmark tasks and compares pass rate, cost and speed.
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
Unit tests for skills: checks that an agent finds and uses your skill. Node 20 and Docker are needed, plus a model key
A skill that guides Claude Code and Codex through writing and fixing pytest API tests with a real-run verification loop.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| Caliper (this card) | no | 228 | |
| Promptfoo | no | 26,908 | compares prompts and models, does not install skills into an agent |
| agent-eval | yes | 277 | evaluates coding agents on your framework and does not break out individual skills |
README in short
The README opens with the claim that one successful try does not prove a skill, and compares success with and without the skill on an inbox-triage example. Next come quick starts for the agent and for the terminal, a walkthrough of the `.eval.yaml` format with three kinds of checks, and `caliper run` output on six tasks. The examples section offers six ready scenarios for common agent failures, and a documentation table leads to the reference for the format, backends and results.