Skill Eval Harness CLI
A Python CLI that measures what a skill changes: runs a case with and without it, lints answer leaks in prompts and diffs reports
- 79 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Runs launch agents on your keys and spend tokens, and script oracles execute code from the manifest
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/skill-eval-harnessDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Runs launch agents on your keys and spend tokens, and script oracles execute code from the manifest
This is third-party code. Review the repository files before installing.
What it does
Skill Eval Harness is a Python command-line tool for testing agent skills. It reads cases from `evals/shared-benchmark.json`, prepares tasks without answer keys, runs them with and without the skill on one model and repetition, then grades results locally and deterministically with no model call in the grade path. The core metric is the lift from the skill: paired comparisons, a confidence interval, tune, holdout and holdback splits, leakage lint and ablations that switch off individual instructions or files of the skill. Three commands ship: `skill-benchmark`, `skill-pi-trigger-eval` and `skill-trigger-matrix`, the last measuring whether an agent loads the skill by itself.
Who it is for. Authors of skills for Claude Code, Codex and other agents who want to prove a skill value with a number and catch regressions after edits.
Good fit when
- You rewrote SKILL.md and want to know whether answers beat the no-skill baseline
- You need to find the instruction or reference whose removal breaks the result
- A CI check should fail when a new edit lowers the share of passed cases
- An agent loads the skill too often or too rarely on its own and you measure it per model
- Before spending budget on a run you want to check that the prompt does not leak the answer
Not a fit when
- You want a general metric set that scores one answer against a rubric: the tool is built for the with-and-without difference
- There is no skill yet: first you need a few real cases in the manifest
- You expect a ready run without Python 3.10 and `uv`
- You need to evaluate product code rather than skill behavior
Example request
Prepare a paired benchmark of my skill, run it with Claude and show the per-case lift and any answer leakage in promptsLimitations
It needs Python 3.10 or newer and `uv`. Runs are made by agents: Claude, Codex, Gemini CLI, Mistral Vibe, Jetty or any runner that writes files per the output contract, and they spend tokens on your keys. Pi support is limited to skill-loading checks and smoke tests. Hidden splits and their storage stay user policy: the tool only prevents mixing all splits by accident. Reading earlier runs after an upgrade needs a separate manifest and telemetry migration per `docs/upgrading.md`.
How to disable. Run `uv tool uninstall skill-eval-harness`. The `eval-runs/` folders and reports in your repositories are separate; delete them by hand.
FAQ
How do I install Skill Eval Harness?
Run `uv tool install skill-eval-harness` to get `skill-benchmark`, `skill-pi-trigger-eval` and `skill-trigger-matrix`. For exact reproducibility pin the version: `uv tool install skill-eval-harness==0.6.0`.
How is it different from general eval frameworks?
General frameworks score one answer against a rubric. Here the same case runs with and without the skill, and the tool spends its surface keeping the difference honest: paired comparisons, splits, leakage lint and ablations.
Does grading need a model?
No, string, regex, file and JSON field checks run locally. A model judge is optional through `--judge-backend` or your own `--judge-cmd`.
How can I tell whether the agent loads the skill itself?
Use `skill-trigger-matrix`: it counts autonomous activations per agent and model pair on cases where the skill should and should not fire.
Is there an example with no model cost?
Yes, `examples/demo-skill/` holds an offline example with a toy skill, ablations and a stub runner with no API.
Discussion
No questions or reviews yet.
Related
CLI toolsA CLI to evaluate and evolve agent skills: declarative YAML cases, runs across engines and structured reports
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
SkillEvaluator
skillevaluator
An NVIDIA framework for evaluating agent skills: quality checks, overlap detection, synthetic checks and live evaluation by an agent. Experimental
A CLI and MCP for testing agent skills: repeatable YAML evals, report comparison, regression detection and skilled-improve
A skill runs a third-party coding-agent harness through 30 benchmark tasks and compares pass rate, cost and speed.
Validates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
README in short
The README explains how the tool differs from ordinary eval systems and gives a table of questions with the commands that answer them. It then covers a five-step core loop, what the CLI owns, installation from PyPI and GitHub and a documentation map.