Evals Skills plugin
Nine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.
- 1,770 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The skill starts a local Python server for the annotation app and reads trace files that may contain user data.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add plugins/evals-skillsDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The skill starts a local Python server for the annotation app and reads trace files that may contain user data.
This is third-party code. Review the repository files before installing.
What it does
A set of nine skills guides an agent through building evals for an AI product: not foundation-model benchmarks but checks for your task. The entry point is `evals-start`, which looks at your situation and routes you to the right skill. `eval-audit` inspects an existing eval pipeline and ranks problems by severity, and `error-discovery` analyzes traces while you do not yet know where the model fails. `error-discovery` takes a JSONL, CSV or JSON file of outputs or traces, works out the data type and builds a single-file review app served by the Python standard library with no dependencies. It then picks a diverse sample of cluster representatives and random examples. You read and leave free-text notes, and the agent sorts them into failure modes, tracks coverage and proposes new samples. The other skills generate synthetic test inputs, write code checks and LLM-judge prompts, calibrate a judge against human labels and evaluate RAG.
Who it is for. Teams that ship an LLM product or agent and want to build evals from real traces rather than general benchmarks.
Good fit when
- Agent traces have piled up but nobody has reviewed them and it is unclear where it fails.
- An eval pipeline exists and an audit with prioritized findings is needed.
- An interface for manual trace annotation is needed without writing an app.
- An LLM judge must be checked against human labels by TPR and TNR.
- A RAG pipeline needs retrieval and generation quality evaluated separately.
Not a fit when
- A public foundation-model benchmark is needed: the skills target evals for a specific product.
- There are no traces or model outputs yet: collect data first, otherwise there is nothing to review.
- Production monitoring, regression suites and cost tracking are needed: the author places them in the course.
Example request
Help me do error analysis on traces.jsonl: build an annotation app, pick diverse samples and group my notes into failure modes.Limitations
The skills cover general eval work and know nothing of your domain, so the authors advise writing your own. Production monitoring, regression suites and cost are not part of the set and belong to the authors' course. The annotation app runs on the Python standard library server and traces are read from a local file. Evals should be written only after error analysis on your data, otherwise the metrics measure the wrong thing. The last commit is from 24 October 2026, and the plugin version is 0.3.1.
How to disable. Remove the skills with `npx skills` or delete the directories from the agent's skills folder and start a new session. For the Claude Code plugin disable it in the `/plugin` window.
FAQ
How do I install the skills?
`npx skills add https://github.com/ai-evals-course/evals-skills`. A single skill is installed with the `--skill error-discovery` flag, and updates are checked with `npx skills check` and `npx skills update`.
Which skill should I start with?
`evals-start`: it determines your situation. With a ready pipeline it routes to `eval-audit`, and with unanalyzed traces to `error-discovery`.
What does error-discovery do?
It reads a JSONL, CSV or JSON file, builds a single-page review app, picks a diverse sample and sorts your notes into failure modes while tracking coverage.
Does the annotation app need dependencies?
No. It is built as one HTML file and served by the standard Python server.
How do I calibrate an LLM judge?
The `validate-evaluator` skill splits data, compares the judge with human labels by TPR and TNR and applies bias correction.
Discussion
No questions or reviews yet.
Related
PluginsSkill set for evaluating an AI product: rubric, dataset, failure codes, code checks and LLM judges with calibration.
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
The run-assert-eval skill turns a description of your agent's risks into tests, a run and a report with trace-cited failures.
TruLens skills walk an agent through LLM app tracing, evaluation setup, diagnosing low scores and runtime blocking guardrails
OpenEval runs coding agents on your tasks with scoring rubrics and records cost, tokens and tool reliability
Validates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
README in short
The authors built the skills from experience helping more than fifty companies and teaching the AI Evals course. The aim of the set is to guard against typical mistakes, such as writing evals before analyzing traces.