Superpowers Evals CLI
Behavioral eval lab Quorum for the superpowers project: drives real agent CLIs and grades them against private criteria
- 145 ★ on the repository
- Last commit
Broad access
Why this level
- Live runs launch agents in no-confirmation modes, and Claude and Codex get broad execution rights
- Some runs copy operator credentials, which can leak into logs if misconfigured
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/superpowers-evalsDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- Live runs launch agents in no-confirmation modes, and Claude and Codex get broad execution rights
- Some runs copy operator credentials, which can leak into logs if misconfigured
This is third-party code. Review the repository files before installing.
What it does
Superpowers Evals is a behavioral eval lab for the superpowers project. The `quorum` tool drives and grades real coding agent CLIs: Claude, Codex, Antigravity, Gemini, Hermes, Kimi, OpenCode, Pi and Copilot. New scenarios use a user-like conversation, then a fresh evidence-based assessment against private criteria plus deterministic post-checks. The authors note the lab checks workflow compliance and is no generic benchmark suite. It checks skill triggering, worktree behaviour, subagent coordination, verification reflexes, review quality and cost-shaping patterns. The `arms` folder describes compared configurations, for example Claude Opus and Codex on branches of pull request 2258. There are two modes. Static checks (`biome`, `tsc`, `bun test`) are safe for public CI and call no models. Live evals launch agents in permissive modes and collect raw transcripts, tool calls, filesystem state and session logs.
Who it is for. Authors of skills and workflows for agents who want to measure process compliance by models, and maintainers of the superpowers project.
Good fit when
- You need to check whether a skill triggers across different agents and models
- Agent configurations are compared before and after a pull request change
- Agent behaviour is graded across a conversation and not a single answer
- Static code checks are needed in public CI without calling models
Not a fit when
- You need a general programming benchmark: the lab checks process compliance
- You have no trusted isolated machine: live evals launch agents in dangerous modes
- You need a simple check of one prompt without scenarios
Example request
Run the quorum static checks without calling models and show which scenarios exist for checking skill triggeringLimitations
Live evals are for trusted maintainers only: Claude runs with `--dangerously-skip-permissions`, Codex with `--dangerously-bypass-approvals-and-sandbox` and Antigravity uses a seeded copy of the operator account. They must not be wired into public CI or keep API keys there. The tool is built on Bun and TypeScript, and the repository is large (over sixteen hundred files). The last commit is dated September 11, 2026.
How to disable. Stop running `quorum run` and delete the cloned repository together with the seeded agent credential copies the runs created.
FAQ
What is Quorum?
The lab tool that drives real agent CLIs and grades them against private criteria and deterministic checks.
Which agents are supported?
Claude, Codex, Antigravity, Gemini, Hermes, Kimi, OpenCode, Pi and Copilot.
Can checks run in public CI?
Only static ones: `biome`, `tsc` and `bun test`. Live runs that launch agents must not be added to public CI.
What does the lab evaluate?
Workflow compliance: skill triggering, worktrees, subagents, verification reflexes, review quality and cost shaping.
Is it a general benchmark?
No, the authors say outright that it is a process compliance lab for superpowers.
Discussion
No questions or reviews yet.
Related
CLI toolsVercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
Runs Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
QA agents, skills and an MCP server: test generation, coverage gaps, flaky tests, security scans and chaos testing
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
A CLI that tests AI agents end to end: derives scenarios from code, runs them live and separates unverified from failed.
Caliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| Superpowers Evals (this card) | no | 145 | |
| Evals Skills | no | 1,770 | scenario-based process checks with real agent CLIs and private criteria |
| SkillEvaluator | yes | 859 | checks agent behaviour in a conversation and does not write graders for one skill |
README in short
The README describes the lab as behavioral checking for superpowers and explains the `quorum` naming. Sections cover the safety model (static and live checks) and the dangers of live runs with the list of permissive flags for each agent.