typesafe-computer-use CLI
A CLI that drives a Mac to a goal from one phrase: code reads the screen, a cheap model picks actions.
- 1,298 ★ on the repository
- Last commit
Broad access
Why this level
- With `--act` the tool drives the real mouse and keyboard, so start with a dry run.
- Screenshots and recognized screen text are sent to the external TypeSafe service, so keep secrets off the screen.
Install
- Works in
Manual install
git clone https://github.com/awlevin/typesafe-computer-useThen `uv sync` and `cp .env.example .env`, with keys entered by the user. macOS 14 or newer, Python 3.12 and terminal screen recording permissions are required.
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- With `--act` the tool drives the real mouse and keyboard, so start with a dry run.
- Screenshots and recognized screen text are sent to the external TypeSafe service, so keep secrets off the screen.
This is third-party code. Review the repository files before installing.
What it does
typesafe-computer-use (jev for short) drives a Mac toward a goal given in a plain phrase for about a fiftieth of a cent per step. The screen is read deterministically: Vision text recognition on the window plus accessibility tree elements. Code computes the facts itself, such as a block's date, the focused field and actions already tried, and hands them over as state. Then one request to TypeSafe's decision model picks the action type, the item and the site, and code executes the action. A large model is called only when text must be written into a field or the classifier has stopped and the screen needs reading. By the author's measurement on one screenshot a decision costs 0.0002 dollars against 0.032 for a large model, and a full step cycle takes about 1.5 seconds. The author notes that reasoning a large model does for free has to be rebuilt here as code. The `clicker "goal"` command makes a one-step dry run by default, and with `--act` it drives the mouse and keyboard for up to 100 steps. Each run writes a `runs/` folder for review. You stop with Ctrl-C or by slamming the mouse into the top-left corner. The project runs on the OSWorld benchmark.
Who it is for. Agent developers who need to drive desktop apps and the browser at a sensible price and can work with a beta.
Good fit when
- You want to see how screenshot-based computer use behaves on a long chain of simple steps.
- An agent must open a site and reach a target page without spending expensive tokens on every step.
- You want to run a scenario in OSWorld and compare price and speed with a large model.
- You need to see where the classifier stopped and review a run from the `runs/` folder.
- The task is made of repetitive UI actions that reduce to choosing from a list.
Not a fit when
- You need reliability on complex sites with unusual layouts, and the author calls the project a beta.
- You have no TypeSafe key or cannot send screenshots and screen text to an external service.
- The computer does not run macOS, with Windows supported only experimentally.
Example request
Open our documentation site in Chrome and reach the pricing page. Do a dry run first, without the --act flag.Limitations
The project is a beta: settings and behavior change between 0.x releases. It needs macOS 14 or newer, Python 3.12, uv, a `TYPESAFE_API_KEY` and terminal permissions for screen recording and accessibility. An Anthropic key is only needed for typing text and the final answer, and the writer can be replaced with LM Studio, Ollama or another compatible server. The terminal is visible on screen and enters recognition, so clear it before a run.
How to disable. Delete the repository clone and the `.env` file with keys, then revoke the terminal's screen recording and accessibility permissions in macOS settings.
FAQ
How much does a step cost?
By the author's measurement about 0.0002 dollars per classifier decision, 155 times cheaper than a large model on the same screenshot. Check current prices with TypeSafe.
What does a dry run do?
One step with no actions: the command prints what it would do. Only the `--act` flag turns control on.
How do I stop a running scenario?
Press Ctrl-C in the terminal or slam the mouse into the top-left corner of the screen.
What is the Anthropic key for?
Only for typing free text, writer-proposed URLs and the final answer. The classifier makes the decisions.
Does it work on Windows?
Windows 10 and 11 are experimental, with details in `docs/windows.md`.
Discussion
No questions or reviews yet.
Related
CLI toolsA local macOS CLI that lets an agent read the screen via OCR, click and type, with no cloud and no vision model
A skill that lets an agent drive API-less native Mac apps with background reads, focus-safe writes and proof for every step.
A Swift MCP server that lets an agent drive any macOS app: click, type and read the interface tree
Mano skill
Mano skill (mano-cua)
A CLI and skill for desktop control from a text task: screenshots, clicks, typing and scrolling, in the cloud or locally on Apple Silicon
A Rust CLI for macOS: an agent reads any app's interface through the accessibility tree and clicks, types and scrolls by element refs, with no screenshots
An agent fills fields and selects menu items in a local application, checking the screen before and after each action.
README in short
The README presents the project as a computer-use loop around a classifier: the classifier picks, code computes facts and only the writer writes. MIT license, Python 3.12.