Autoresearch for skills
autoresearch
A skill that improves another skill: runs it on test inputs, scores outputs with binary checks and mutates the prompt
- 980 ★ on the repository
- Last commit
Extended access
Why this level
- Runs another skill repeatedly and rewrites its SKILL.md
- Can burn many tokens without a budget cap
Install
- Works in
Manual install
git clone https://github.com/olelehmann1337/autoresearch-skill ~/.claude/skills/autoresearchThen create ~/.claude/skills/autoresearch/references/ and move eval-guide.md into it.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Runs another skill repeatedly and rewrites its SKILL.md
- Can burn many tokens without a budget cap
This is third-party code. Review the repository files before installing.
What it does
The skill adapts Andrej Karpathy's autoresearch approach from model training to Claude Code skill prompts. It first collects the target SKILL.md path, 3-5 test inputs and 3-6 yes or no checks from the user. It then loops: runs the skill, scores every output, applies one prompt mutation and keeps it only if the score improves. The output is an improved SKILL.md, a results.tsv log, a changelog of every attempt and an HTML dashboard to watch progress.
Who it is for. For skill authors whose skill works inconsistently and who want measurable quality gains.
Good fit when
- A skill sometimes produces bad output and it is unclear what to fix
- You want to compare prompt versions against the same checks
- You need a binary eval suite for a skill
Not a fit when
- Output quality cannot be reduced to yes or no checks
- Token budget is limited: the loop runs the skill dozens of times
Example request
Run autoresearch on my skill at ~/.claude/skills/newsletter/SKILL.md with a 10-cycle budgetLimitations
There is no README; all documentation is in SKILL.md and eval-guide.md. By default the loop has no iteration cap and uses many model calls, so set a budget first. SKILL.md links to references/eval-guide.md while the file sits at the repository root, so move it when installing.
How to disable. Delete the autoresearch folder from ~/.claude/skills/.
FAQ
Why only binary checks?
Rating scales are noisy between runs, while yes or no gives a stable score for comparing mutations.
How many runs per mutation?
Five by default; you can change it before starting.
Discussion
No questions or reviews yet.
Related
SkillsDarwin Skill
darwin-skill
A skill that improves other skills: scores SKILL.md on nine criteria, edits, tests with prompts and reverts
A skill that designs, tests, evaluates and packages other agent skills, starting from architecture
A skill for creating skills that proves them by behavior: before-and-after runs, regression evals and ecosystem audit
Self-learning
self-learning-skills
A meta-skill that saves a hard-won working procedure as a new skill or rule, so the next session starts already knowing it
A skill that helps an agent author other skills to the agentskills.io spec: structure, metadata, progressive disclosure and validation
A CLI and Python SDK to search, install, create and evaluate reusable agent skills from a shared library
README in short
The repository has no README. SKILL.md covers collecting context before starting, building the eval suite, a baseline run, the mutation loop and the output file format. eval-guide.md explains how to write good binary checks and which mistakes to avoid.
README badge
Original SKILL.md
--- name: autoresearch description: "Autonomously optimize any Claude Code skill by running it repeatedly, scoring outputs against binary evals, mutating the prompt, and keeping improvements. Based on Karpathy's autoresearch methodology. Use when: optimize this skill, improve this skill, run autoresearch on, make this skill better, self-improve skill, benchmark skill, eval my skill, run evals on. Outputs: an improved SKILL.md, a results log, and a changelog of every mutation tried." --- # Autoresearch for Skills Most skills work about 70% of the time. The other 30% you get garbage. The fix isn't to rewrite the skill from scratch. It's to let an agent run it dozens of times, score every output, and tighten the prompt until that 30% disappears. This skill adapts Andrej Karpathy's autoresearch methodology (autonomous experimentation loops) to Claude Code skills. Instead of optimizing ML training code, we optimize skill prompts. --- ## the core job Take any existing skill, define what "good output" looks like as binary yes/no checks, then run an autonomous loop that: 1. Generates outputs from the skill using test inputs 2. Scores every output against the eval criteria 3. Mutates the skill prompt to fix failures 4. Keeps mutations that improve the score, discards the rest 5. Repeats until the score ceiling is hit or the user stops it **Output:** An improved SKILL.md + `results.tsv` log + `changelog.md` of every mutation attempted + a live HTML dashboard you can watch in your browser.