ClawBench CLI
A benchmark that tests browser agents on 281 everyday tasks across live websites and records each run in five layers.
- 1,278 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The agent browses live sites and fills forms, so interception of the final request matters: do not turn it off.
- Model keys are passed into the container, so keep them separate from production ones.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/clawbenchDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The agent browses live sites and fills forms, so interception of the final request matters: do not turn it off.
- Model keys are passed into the container, so keep them separate from production ones.
This is third-party code. Review the repository files before installing.
What it does
ClawBench is an open benchmark for browser AI agents, published in Findings of EMNLP 2026. Tasks come from everyday life: booking travel, ordering food, applying for a job, sorting email. V1 has 152 tasks on 143 sites, V2 has 129 tasks on 63 sites, for 281 tasks across 163 live websites and 15 categories. Each run starts an isolated Docker or Podman container with Chromium and the agent. An interceptor records actions in five layers: screen recording, requests, actions, agent messages and interception data. Scoring has two stages: first a deterministic check that the final HTTP request matched the task's URL and method, then an LLM judge grades the reply. A leaderboard publishes results for different models and harnesses. You run it through the `clawbench-eval` package from PyPI: the `clawbench` command opens a terminal interface where you add a model and pick a task. For agent developers there is reproduction of published results, run traces in a Hugging Face dataset and a guide to adding your own tasks.
Who it is for. Developers and researchers of browser agents who need to compare a model or harness on real tasks instead of synthetic pages.
Good fit when
- You are choosing a model for a browser agent and want to compare it on everyday tasks against a public leaderboard.
- You need to reproduce a published result from run traces.
- You build your own agent harness and look for tasks where it breaks.
- You want to add a new test, since most contributions are a single JSON file.
- You need screen and request recordings to analyze agent mistakes.
Not a fit when
- You need a tool for your agent's work, whereas ClawBench only measures.
- You have no Docker or Podman, and the container is required.
- You need stable tests on fixed pages, while the sites are live and change.
Example request
Run ClawBench with my model on one V2 task without judge scoring and show the run recording.Limitations
Python 3.11, uv and a working Docker or Podman are required. Tasks run on live websites, so results depend on site changes and are described in the FAQ as limitations. The default judge needs a model key, and a run without scoring is possible. The authors note that in the historical V1 test the strongest agent completed about a third of tasks. Some tasks need accounts on the sites, with the procedure in the project's FAQ. Interception stops the final request, but watch the model's token spend.
How to disable. Run `uv tool uninstall clawbench-eval`, stop and remove the Docker or Podman containers created by runs, and delete the results folder.
FAQ
What does ClawBench measure?
A browser agent's ability to finish an everyday task on a live site. The first stage checks the final HTTP request, the second gives the reply to an LLM judge.
How many tasks does it have?
281 tasks across 163 live sites: 152 in V1 and 129 in V2, in 15 life categories.
What is recorded during a run?
Five layers: video, requests, actions, agent messages and interception data.
Can I add my own task?
Yes, most contributions are a single JSON file, see `CONTRIBUTING.md`.
Where do I see results?
On the claw-bench.com leaderboard and on Hugging Face, which also holds traces for reproduction.
Discussion
No questions or reviews yet.
Related
CLI toolsOpenEval runs coding agents on your tasks with scoring rubrics and records cost, tokens and tool reliability
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
Runs Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it
One binary for agent browser automation: a CLI with browser and check skills, an MCP server and clients for JS, Python and Java
CLI checks a web page in Chromium: finds every control, uses it and reports what broke. No tests to write and no LLM.
An MCP server that spins up several browser agents to test a site for broken links, UI bugs and accessibility issues
README in short
The authors describe ClawBench as a browser-agent benchmark on everyday tasks with five recording layers and two-stage scoring. Apache-2.0 license, with a paper, a leaderboard and documentation.