labrat CLI
A local runtime where Claude Code or Codex run research for hours: idea branches compete for budget and the verified ones win
- 237 ★ on the repository
- Last commit
- MIT
Broad access
Why this level
- The agent works for hours with permissions to run Python scripts and writes lab files
- Experiments consume compute and may call paid external services if you connect them
- An unattended loop needs boundaries set in advance: a narrow permission list and stop criteria
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/labratDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The agent works for hours with permissions to run Python scripts and writes lab files
- Experiments consume compute and may call paid external services if you connect them
- An unattended loop needs boundaries set in advance: a narrow permission list and stop criteria
This is third-party code. Review the repository files before installing.
What it does
labrat puts a real research problem in front of Claude Code or Codex and supplies the frame in which an agent works for hours: a scoreboard, an experiment queue and one evaluator. The search is population-based rather than single-threaded. Families of hypotheses compete for compute budget, and the ones that produce reproducible progress get more room. You define the problem and a baseline, the agent explores families of ideas, the runtime keeps the queue moving and the evaluator scores results consistently. The design splits into a supervisor and workers. The agent supervises while bounded tasks such as probes, mutations, crossover and audit run in workers. State sits in files and logs under the File-as-Bus principle, so the supervisor keeps thin control over thick project state. A family gains standing when it wins a hard held-out challenge (`prediction_tests`) rather than merely fitting the local metric. The `labrat` command works from a repository clone: `doctor`, `bootstrap`, `status`, `next-prompt`, `new`, `check-readiness`. Labs are created from a profile (currently `transformer-arch`) or from scratch. Each lab has `AGENTS.md` for Codex, `CLAUDE.md` and slash commands in `.claude/commands/` for Claude Code: `/next`, `/why-stuck`, `/synthesize`, `/audit-candidate`, `/frame-break`, `/consolidate`. The dashboard opens through `python -m http.server`.
Who it is for. Machine learning researchers and engineers with a measurable problem and a baseline who want to leave an agent running overnight under strict evaluation rules.
Good fit when
- You tune the architecture or hyperparameters of a small model and the evaluation is fixed in advance and reproducible
- You iterate on prompts and rubrics against a fixed test set
- Several search or ranking strategies must be compared so one wins on a hard slice rather than only on average
- An agent runs for hours and you want a scoreboard, a queue and the reasons for a stall through `/why-stuck`
- You want a lab template: the `transformer-arch` profile scaffolds a runnable frame in one command
Not a fit when
- The problem has no baseline and no bounded runnable experiment: the runtime is built around the evaluator
- You need a short one-off run: Phase 0 with a brief, sources and evaluation configs is too heavy
- The metric cannot be computed consistently: the working loop relies on external scoring of results
- You cannot give the agent a limited set of permissions to run scripts
Example request
Scaffold a lab from the transformer-arch profile, run labrat doctor and show the current phase prompt with next-prompt and runner claudeLimitations
- One ready profile is supported, `transformer-arch`, and it uses a synthetic runner that you replace with your own training script in `scripts/run_experiment.py`. Profiles for world models and multiple datasets are listed as upcoming.
- To run without prompts the agent gets a narrow permission list in the lab's `.claude/settings.json`. A human makes that edit and the agent does not write it. Dangerous operations such as `rm` and `git push` keep asking for consent.
- Installation is an editable install from a repository clone: `labrat` takes templates and scripts from that folder.
- The last commit dates from August 2026. The Chinese README may lag behind the English one.
How to disable. Stop the lab processes and delete the lab folder. The editable install comes off with pip uninstall labrat, and the virtual environment goes with the .venv folder.
FAQ
How do I run labrat in five minutes?
Create a virtual environment, run `pip install -e '.[nlp-sentiment]'` from the repository clone, then `labrat doctor` and `labrat bootstrap` with `--lab-dir examples/nlp-sentiment/research_lab`. `python -m http.server 8787` opens the dashboard.
How do I use labrat with Claude Code?
Open Claude Code in the lab folder and type `/next`: it prints the current phase prompt and executes it. You can also get the prompt by hand with `labrat next-prompt --runner claude`.
Does labrat work with Codex?
Yes, with the same file contract. The agent reads `AGENTS.md` and runs `next-prompt` with `--runner codex`, and the `labrat-operator` skill loads optionally.
What does winning a held-out challenge mean?
A family of ideas gains standing when it wins a hard test that is not part of the local search metric. Such checks are recorded in `evaluation.yaml` as `prediction_tests`.
How do I create my own lab?
With `labrat new my_lab`, then `labrat doctor` and the design phase prompt. Phase 0 must produce `branches.yaml`, `dead_ends.md`, `research_brief.md`, `research_sources.md`, `evaluation.yaml` and `runtime.yaml`.
Discussion
No questions or reviews yet.
Related
CLI toolsA research system on top of Codex CLI or Claude Code: runs of experiments, discussion of progress and a paper draft.
A personal ~/.claude for Claude Code: twelve subagents, review commands, worktree and goal skills and a Go status line.
A local whiteboard and MCP server for research agents: every claim is tied to the run and the code that produced it
LACP runs Claude Code, Codex and Hermes through local policy gates, budgets and an evidence log, and rolls back failed changes
A Claude Code plugin with hooks and 26 skills: measures what the agent knows, blocks edits until code is read, keeps memory
A Rust CLI that builds a knowledge graph of code and decisions and feeds the agent the right slice through Claude Code hooks
README in short
The page explains the idea of population search and lists the principles: asynchrony, funding of families, external evaluation, the supervisor and worker model, the File-as-Bus workspace and decisive challenges. Then come a five-minute run, profiles, creating a lab from scratch and a document map. Background: the runtime grew from internal research on world models at DXRG.