skilled CLI
A CLI and MCP for testing agent skills: repeatable YAML evals, report comparison, regression detection and skilled-improve
- 72 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- Runs launch your engine (Codex, Claude Code and others) and spend account limits and money
- The skilled setup command writes workflow files into the project, and MCP with the --allow-execution flag lets the agent start evaluations
Install
- Works in
Manual install
curl -fsSL https://github.com/DeadHead04/skillEd/releases/latest/download/install.sh | bashmacOS and Linux. Then add ~/.local/bin to PATH. On Windows use install.ps1 from the release.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Runs launch your engine (Codex, Claude Code and others) and spend account limits and money
- The skilled setup command writes workflow files into the project, and MCP with the --allow-execution flag lets the agent start evaluations
This is third-party code. Review the repository files before installing.
What it does
A skill can look right and still fail on edge cases. skilled turns expected behavior into repeatable checks so you can judge a change to the skill, agent or model before relying on it. One install includes the evaluation CLI, the MCP tools and the `skilled-improve` workflow. It uses your skills and your local engine login, and the release bundles its own Node. For a skill with `SKILL.md` and `evals/eval.yaml`, `skilled validate` checks files without calling a model, `skilled run --dry-run` shows the selected cases and `skilled run` runs the configured engine and saves reports in JSON, HTML and JUnit. After the skill is revised the same cases run again, and `skilled compare ... --fail-on-regression` returns a nonzero code on a regression. Checks include scripts, assertions and model judges, multi-turn cases and repeated runs. Engines are Codex, Claude Code, OpenCode, Qoder CLI, Qwen Code and custom ones, with native, Docker or OpenSandbox execution. The `skilled setup --host codex` command (or `claude-code`) enables the `skilled-improve` workflow in a project: the agent adds boundary cases, saves a baseline, improves the skill and rechecks on independent holdouts. The `skilled mcp --workspace` command starts an MCP server over stdio with `skilled_start`, `skilled_status`, `skilled_results`, `skilled_compare` and `skilled_cancel`, and execution is enabled by the `--allow-execution` flag. The author shows an example: a refund-policy skill passed 13 of 18 cases, after revision 18 of 18 and 10 of 10 holdouts.
Who it is for. Skill authors and teams who maintain skill sets for Codex and Claude Code and want to catch regressions when a skill, agent or model changes.
Good fit when
- A skill was edited and you need to confirm it did not break boundary cases and invalid inputs
- You need saved reports before and after an edit with an exit code for regressions in CI
- You want to measure a skill contribution with and without it in isolated Docker
- The agent should add tests, improve the skill and recheck on independent cases itself
Not a fit when
- You need continuous monitoring of conversations: comparison works on saved runs
- You need an accuracy guarantee from one example: the author notes it covers one skill and engine
- You need install through npm or Homebrew: distribution is through GitHub Releases only
Example request
Use skilled-improve on my skill at ./skills/refund-policy: add boundary cases, save a baseline, improve the skill and recheck on independent holdouts.Limitations
Distribution is through GitHub Releases, and there is no npm package or Homebrew formula. The installer verifies the SHA-256 checksum of the archive. Native checks use your existing Codex or Claude Code login, and model usage follows the limits and billing of your account. Compare reports with the warnings in mind: reports lack complete input fingerprints, so matching cases alone say little about equivalent grading conditions.
Benchmarks with and without the skill require Docker, and native and OpenSandbox modes are rejected for them. The author treats authenticated Codex inside Docker as unverified. Execution through MCP is enabled by the `--allow-execution` flag, and the approval policy of your host still applies. The license is Apache 2.0 and the last commit is from October 30, 2026.
How to disable. Run `skilled setup --host codex --remove` (or with `claude-code`) to remove an unmodified managed copy of the workflow. The CLI itself is removed following the installation guide, and saved reports live in your `reports/` directories and are deleted by hand.
FAQ
How do I install skilled?
On macOS and Linux run `curl -fsSL https://github.com/DeadHead04/skillEd/releases/latest/download/install.sh | bash`, add `~/.local/bin` to PATH and check `skilled --version`. For Windows there is an `install.ps1` script, and the installer verifies SHA-256.
How do I check a skill without calling a model?
Run `skilled validate --skill .` and `skilled run evals/eval.yaml --dry-run`: files and selected cases are checked and no model is called.
How do I catch a regression between skill versions?
Run `skilled run` before and after an edit into different report directories, then `skilled compare reports/before/iteration-1/report.json reports/after/iteration-1/report.json --fail-on-regression`. The command returns a nonzero code on a regression.
How do I connect skilled through MCP?
Start `skilled mcp --workspace /absolute/path/to/your-project` as a local stdio server. By default validation, dry runs and comparison are available, and running evaluations is enabled with the `--allow-execution` flag.
What is skilled-improve?
It is an agent workflow with no separate terminal command. After `skilled setup --host codex` the agent adds cases, saves a baseline, revises the skill and rechecks it on independent cases when you ask.
Discussion
No questions or reviews yet.
Related
CLI toolsA CLI to evaluate and evolve agent skills: declarative YAML cases, runs across engines and structured reports
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
Validates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
A Python CLI that measures what a skill changes: runs a case with and without it, lints answer leaks in prompts and diffs reports
Unit tests for skills: checks that an agent finds and uses your skill. Node 20 and Docker are needed, plus a model key
The QASkills.sh directory MCP server: an agent searches, reads and installs testing skills without leaving the conversation.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| skilled (this card) | no | 72 | |
| Skill Eval Harness | no | 79 | a test harness for skill evaluation, while this one adds saved-report comparison, an MCP server and a skill improvement workflow |
| SkillEvaluator | yes | 859 | evaluates skills, while this one tracks regressions between versions, supports several engines and produces reports for CI |
README in short
The description explains the purpose of skill checks, installation, connecting to a project and the evaluation loop with commands. It then gives an example of improving a refund-policy skill with caveats, a capability table, connecting through MCP, documentation and release status.