SanityHarness CLI
An evaluation harness for coding agents: 26 compact tasks in 6 languages in Docker containers, weighted score and integrity checks
- 244 ★ on the repository
- Last commit
Extended access
Why this level
- Runs the chosen agent in Docker containers where it can execute any commands
- A run spends the tokens and limits of the chosen agent and model
- Agents get access to your API keys if you pass them into the container
Install
- Works in
Manual install
git clone https://github.com/lemon07r/sanityharness.git
cd sanityharness
make tools
make buildNeeds Go 1.25 and Docker; bubblewrap is optional for agent isolation. Prebuilt binaries are in the releases.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Runs the chosen agent in Docker containers where it can execute any commands
- A run spends the tokens and limits of the chosen agent and model
- Agents get access to your API keys if you pass them into the container
This is third-party code. Review the repository files before installing.
What it does
SanityHarness runs a coding agent through 26 small but hard tasks in Go, Rust, TypeScript, Kotlin, Dart and Zig. Each task goes into its own Docker container: it starts with `sleep infinity` and commands run through `docker exec`, so containers are reused. Scores are weighted by empirically derived difficulty factors, and the result comes with a detailed report. 19 agents are built in, among them Gemini, Claude, OpenCode, Codex and Goose. `sanity eval --agent gemini` starts an evaluation, and the flags `--tier all`, `--parallel 4` and `--dry-run` choose the task set, concurrency and a preview. Hidden tests apply only during evaluation, and a BLAKE3 check confirms the integrity of submitted results. `sanity run` with `--watch` reruns tests on file changes, and `sanity init` creates a workspace with stubs. Where bubblewrap is available the agent is isolated to its workspace, and build caches in `.sanity-cache/` speed up repeat runs.
Who it is for. Teams and researchers who compare coding agents and models on identical tasks and want a reproducible evaluation with protection against gaming.
Good fit when
- You need to compare Claude, Codex and Gemini on one task set with difficulty weights
- You maintain a leaderboard and BLAKE3 checks protect submitted data
- You want a parallel run of 4 tasks with a shared report
- The agent has to run in isolation so it sees only its own workspace
- Before launching you want to see the evaluation plan with `--dry-run`
Not a fit when
- You need a check on your own repository: the set is 26 teaching tasks, not your code
- Docker is unavailable: every task runs only in a container
- You need evaluation in other languages: the set has six
Example request
Run sanity eval for the claude agent on the core task set with --parallel 4 and show the final scoreLimitations
It needs Go 1.25 and a running Docker daemon, and bubblewrap optionally adds agent isolation. Building from source goes through `make build`, and prebuilt binaries are in the releases. The set is small, 26 tasks across six languages, so a result reflects only this set. Version 1.8 is live in releases and on the leaderboard, and to compare with older data the author suggests version 1.6.1 with the `--legacy` flag. The last commit is from October 2026.
How to disable. Delete the built sanity binary and the repository folder, stop and remove the Docker containers with docker rm, and delete the `.sanity-cache/` caches in workspaces by hand.
FAQ
What is in the task set?
26 tasks in Go, Rust, TypeScript, Kotlin, Dart and Zig, divided by tier and difficulty. See the list with ./sanity list and the --language, --tier and --difficulty filters.
Which agents are supported?
19 are built in, among them Gemini, Claude, OpenCode, Codex and Goose. Pick one with --agent and set the model with --model.
What are hidden tests for?
They apply only during evaluation, so an agent cannot tailor a solution to known checks. The integrity of submitted results is confirmed with a BLAKE3 hash.
Is Docker required?
Yes, each task runs in an isolated container. Bubblewrap can be installed for extra agent isolation.
How do I solve one task by hand?
Create a workspace with stubs using ./sanity init go/bank-account, then ./sanity run go/bank-account runs the tests, and with --watch reruns them on file changes.
Discussion
No questions or reviews yet.
Related
CLI toolsRuns Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
A CLI that rates the maturity of an AI agent harness in a repository from L0 to L4: 36 file checks with no model calls or network.
A Harbor fork for evaluating coding agents in sandboxes: Claude Code, Codex and others run inside tasks with no internet.
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
A benchmark that tests browser agents on 281 everyday tasks across live websites and records each run in five layers.
Agent Reliability Analyzer statically scans AI agent code, finds reliability gaps with 340 plus rules and applies fixes
README in short
The description calls the project a lightweight evaluation harness for coding agents, lists features and a quick start: dependencies, build, global flags. It then covers the list, init, run and eval commands, available tasks, configuration, built-in agents and how containers work.