Reforge AI Evals plugin
Skill set for evaluating an AI product: rubric, dataset, failure codes, code checks and LLM judges with calibration.
- 98 ★ on the repository
- Last commit
- MIT
Minimal access
Why this level
- The skills work with plain project files: markdown, CSV, JSON and HTML.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add plugins/reforge-ai-evalsDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewMinimal accessThe tool runs nothing and makes no network calls.
- The skills work with plain project files: markdown, CSV, JSON and HTML.
This is third-party code. Review the repository files before installing.
What it does
Nine skills guide an agent through the eval workflow. The eval-guide skill checks where you are and loads the right one. From there the agent writes a rubric into the PRD (eval-rubric), builds a diverse dataset from a User Input Grid (uig), clusters your notes into named failure codes (trace-codes), writes code checks and LLM judge prompts, measures a judge against your labels with TPR and TNR (llm-align) and turns support tickets into regression dataset rows (ticket-to-eval). Reading traces and labeling a sample stay with the human: the skills prepare everything around these two steps.
Who it is for. Product managers and AI feature engineers who want to measure answer quality with data and labels.
Good fit when
- An AI feature PRD has no rubric that defines a good answer.
- You need test inputs that cover rare and borderline cases.
- An LLM judge disagrees with human labels and you need agreement metrics.
- Bad production traces should become regression examples.
Not a fit when
- You need to run evals over a dataset: the eval-run skill is listed as coming next.
- There is nothing to evaluate because the product has no traces or user flows.
- You need a platform with dashboards and production monitoring.
Example request
Write a v1 eval rubric from my PRD for a product-search assistant.Limitations
The skills give structure and drafts, but the key steps stay with you: reading traces in your own words and labeling 20–40 outputs as pass or fail, otherwise a judge cannot be calibrated. Running the agent and scorers over a dataset, multi-step trace analysis and production monitoring are announced as upcoming skills. The trace-review-app skill is marked as a draft. The material is based on the Reforge AI Evals course, and a content license sits in a separate LICENSE-CONTENT file.
How to disable. Remove the plugin with `claude plugin uninstall ai-evals@calibre-labs` or delete the skill folders from the agent directory. Rubrics and datasets already created stay in the project as plain files.
FAQ
Which skill do I start with?
With eval-guide. It looks at what the project already has and loads the next suitable skill.
What stays with the human?
Reading traces and labeling a sample: 20–40 outputs by hand, about half pass and half fail. Judge TPR and TNR are computed from this labeling.
Do I need an eval platform account?
No. The skills read and write plain files: markdown, CSV, JSON and HTML.
When is a code check better than an LLM judge?
For structural and deterministic properties: counts, required sections, citations, format. A judge fits meaning: correctness, tone, edge-case handling.
How do I install it in Claude Code?
With `claude plugin marketplace add Calibre-Labs/reforge-ai-evals` and `claude plugin install ai-evals@calibre-labs`. Codex has its own plugin manifest.
Discussion
No questions or reviews yet.
Related
PluginsNine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
A Python SDK and CLI that evaluate agent self-improvement on ready benchmarks: a candidate goes in, scores come back to your loop.
TruLens skills walk an agent through LLM app tracing, evaluation setup, diagnosing low scores and runtime blocking guardrails
A Microsoft suite for RAG testing: it generates questions for a dataset and scores answers with a judge model, plus three skills
Runs Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
README in short
The repository packages the Reforge AI Evals course as skills with templates: ready PRDs, v1 and v2 rubrics, annotated traces for three sample agents and results pages with scorer examples.