RSI Eval CLI
A Python SDK and CLI that evaluate agent self-improvement on ready benchmarks: a candidate goes in, scores come back to your loop.
- 470 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- An evaluation runs candidate code and benchmark tasks in Docker containers.
- Runs consume tokens and are billed by the model provider.
Install
- Works in
Manual install
python -m pip install -e '.[harbor]'Run inside the cloned repository; then check Docker and run `rsi-eval list`.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- An evaluation runs candidate code and benchmark tasks in Docker containers.
- Runs consume tokens and are billed by the model provider.
This is third-party code. Review the repository files before installing.
What it does
RSI Eval gives an evaluation boundary to researchers of recursive self-improvement methods. You build the loop: candidate generation, memory, selection and stopping rules stay yours and you call `evaluate()` from your code. The SDK runs pinned tasks and their graders and returns per-task scores, execution status, token use, verifier output and trajectory paths. Each evaluation snapshots the candidate and task identities so an experiment can resume after a failure. The catalog holds 24 evaluation configurations and two labeled legacy subsets: Terminal-Bench 2.0 and 2.1, SWE-bench Verified, Multilingual and Pro, Aider Polyglot, LiveCodeBench, MMLU, GPQA, ARC-AGI, MGSM, DROP, FiNER, LawBench, EngDesign, JobBench and others. `rsi-eval list` prints exact versions and task counts. The package includes a Codex candidate template and four method integrations, Meta-Harness among them.
Who it is for. Agent researchers who develop self-improvement methods and want to run candidates on SWE-bench and Terminal-Bench with no homemade runner.
Good fit when
- An agent self-improvement method must be tested on SWE-bench Verified and Terminal-Bench through one interface.
- Per-task scores, token use and trajectories in a common format are wanted from your own loop.
- An experiment stopped on a failure and must resume from a saved state.
- A ready candidate template and a Meta-Harness integration are needed for Codex tasks.
Not a fit when
- A single run of one benchmark with no self-improvement loop is needed.
- There is no Docker, which the default task backend needs.
- A ready optimizer is expected, and the SDK contains none.
Example request
Show `rsi-eval list`, pick three SWE-bench Verified tasks and run the current candidate through `evaluate()` with a token log.Limitations
Running evaluations needs the `harbor` extra (`python -m pip install -e ".[harbor]"`), which pins Harbor 0.18.0 and requires its executable on PATH. Tasks run in Docker by default: Linux needs Docker Engine with Compose and Buildx and macOS needs Docker Desktop or Colima. Method repositories and benchmark assets are fetched separately during preparation. Many configurations need model keys, and large benchmark runs consume tokens. The last commit in the repository is from October 2026.
How to disable. Run `pip uninstall rsi-eval` and delete the experiment folders together with candidate snapshots.
FAQ
How do I install RSI Eval?
From a cloned repository: `python -m pip install -e ".[harbor]"`. The `harbor` extra pins Harbor 0.18.0 and its executable must be on PATH.
Which benchmarks are supported?
Terminal-Bench 2.0 and 2.1, SWE-bench Verified, Multilingual and Pro, Aider Polyglot, LiveCodeBench, MMLU, GPQA, ARC-AGI, MGSM, DROP, FiNER, USPTO, LawBench, EngDesign-Open, JobBench, Harvey LAB, APEX and others.
Do I need my own optimizer?
Yes. The SDK sets the evaluation boundary, and you write the self-improvement loop, memory and stopping rules yourself. It has no optimizer base class.
What does evaluate() return?
Per-task scores, execution status, token use, verifier output and trajectory paths in a common format. The candidate snapshot is saved for recovery.
Where do tasks run?
In Docker by default. Linux needs Docker Engine with Compose and Buildx and macOS needs Docker Desktop or a Colima VM.
Discussion
No questions or reviews yet.
Related
CLI toolsA Python CLI that measures what a skill changes: runs a case with and without it, lints answer leaks in prompts and diffs reports
Nine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.
OpenEval runs coding agents on your tasks with scoring rubrics and records cost, tokens and tool reliability
A Microsoft suite for RAG testing: it generates questions for a dataset and scores answers with a judge model, plus three skills
Skill set for evaluating an AI product: rubric, dataset, failure codes, code checks and LLM judges with calibration.
Runs Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
README in short
One evaluation interface for agent self-improvement methods: 24 benchmark configurations, a common result format and recovery after a failure.