ASSERT skill
The run-assert-eval skill turns a description of your agent's risks into tests, a run and a report with trace-cited failures.
- 362 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Conversations and system descriptions go to the chosen model providers and runs consume paid tokens
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/assertDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Conversations and system descriptions go to the chosen model providers and runs consume paid tokens
This is third-party code. Review the repository files before installing.
What it does
ASSERT, a local package for evaluating AI agents and LLM apps, takes requirements from text: product conditions, policies, a system prompt. From them it derives behavior categories and single- and multi-turn test conversations, runs them against the target, and an LLM judge scores each conversation against the policies. The `run-assert-eval` skill in `.claude/skills/` drives this path in chat: it gathers a risk from the user or through the Clarity MCP server, researches how that risk was evaluated in the literature, writes an `eval_config.yaml`, runs the pipeline once approved and reports violation rates with examples.
Who it is for. Developers and testers of agents and chatbots who have requirements and policies but no systematic checks that they hold.
Good fit when
- You need to check that a support bot never gives legal advice.
- You have a product description or threat model and want tests derived from it.
- You want a violation rate and concrete failing conversations.
- Before release you need a repeatable set of agent behavior checks.
Not a fit when
- You need to fix the agent; the skill finds and reports failures, it does not repair them.
- Checking a few phrases by hand is enough.
- Conversations cannot be sent to an external judge model.
Example request
Evaluate my support agent for budget violations: describe the risk, propose a config, wait for approval and run the pipeline.Limitations
Python 3.11 or newer and model access for generation and judging are required, which costs money. Risk discovery through Clarity needs Python 3.12 and an MCP-capable environment such as Claude Code, Cursor or VS Code with Copilot. The judge's score remains a model's score and needs spot checks. Test quality depends on how complete the requirements are: vague policies produce vague checks. If something misbehaves, check the README and the repository's open discussions.
How to disable. Remove `.claude/skills/run-assert-eval` from the project and the Python package from the environment. Run outputs live in `artifacts/`; delete them separately.
FAQ
Where do risks come from?
From Clarity, the recommended path, or from the user: a description, PRD, threat model or incident report.
What is in the report?
Pass and violation rates per behavior, and failing examples with trace references.
Is Clarity required?
No. The README calls it recommended, not required.
What about locality?
The package is described as local-first and not tied to an agent framework. Model calls still go to the provider you choose.
Which targets can be tested?
README descriptions include callables and other targets; the list is on the `docs/targets/callable.md` page.
Discussion
No questions or reviews yet.
Related
SkillsNine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
agentevals scores AI agent behavior from OpenTelemetry traces and ships MCP tools plus the /eval and /inspect skills.
Validates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
A CLI that tests AI agents end to end: derives scenarios from code, runs them live and separates unverified from failed.
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
README in short
The skill leads the user from a risk description to an approved config and a report, testing one behavior at a time.