agent-desktop CLI
A Rust CLI for macOS: an agent reads any app's interface through the accessibility tree and clicks, types and scrolls by element refs, with no screenshots
- 1,829 ★ on the repository
- Last commit
- Apache-2.0
Broad access
Why this level
- Sees and presses any element in any app, can force-quit them, type text and send messages in your name
- Reads and writes the clipboard and reads notifications, which can hold codes and private messages
- Asks macOS for Accessibility and Screen Recording, and the permission goes to the whole terminal
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/agent-desktopDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- Sees and presses any element in any app, can force-quit them, type text and send messages in your name
- Reads and writes the clipboard and reads notifications, which can hold codes and private messages
- Asks macOS for Accessibility and Screen Recording, and the permission goes to the whole terminal
This is third-party code. Review the repository files before installing.
What it does
agent-desktop is a Rust command-line program through which an agent sees macOS windows and controls them. The program makes no decisions: the look, choose, act loop belongs to the agent, and `agent-desktop` answers every command with structured JSON carrying an error code and a retry hint. It reads the interface through the system accessibility tree, so no screenshots or pixel matching are needed. The `snapshot --app Finder -i` command returns the window's element tree and gives buttons, fields and menu items refs such as `@s8f3k2p9:e1`. `click`, `type`, `set-value`, `select`, `toggle`, `scroll` and `drag` act on a ref, and `press cmd+s` sends a key combination. Before acting the program checks that the element is visible, enabled and supports the action. If the interface changed, it returns `STALE_REF` or `AMBIGUOUS_TARGET` instead of clicking blindly. For dense apps such as Slack or VS Code the `--skeleton` mode gives a short overview and `--root` expands the region you need. Beyond elements there are commands to launch and close apps, move windows, read and write the clipboard, read notifications, wait for an element, text or window, and run steps together with `batch`. `session start` keeps a log of steps that `trace export` assembles into a single HTML file. `launch "Obsidian" --cdp` opens a debugging port on `127.0.0.1` for a Chromium app so Playwright or `agent-browser` can drive its web contents. The `agent-desktop` skill in `skills/` explains these commands to the agent.
Who it is for. For agent developers and Claude Code or Codex users on a Mac who want an agent to drive ordinary apps that have no API: Finder, System Settings, Slack, Xcode.
Good fit when
- You want to automate a Mac app that has no command line or API, such as a setting in System Settings or a step in Xcode
- An agent working from screenshots misses buttons and you need repeatable actions through stable element refs
- An Electron app such as Slack or Obsidian has to be read cheaply in tokens: the `--skeleton` mode shortens the interface snapshot
- You want to review steps after a run: a `session` log with `trace export` shows what the agent did in every window
Not a fit when
- You run Windows or Linux: the platform table marks those systems as planned, and only macOS is ready
- You need a web browser: for pages tools like Playwright fit better, while `agent-desktop` handles native windows and menus
- You would rather not install Rust: building from source needs Rust 1.89 and macOS 13.0, while the npm package brings a prebuilt binary
- The agent has to hold a key or mouse button: the `key-down`, `key-up`, `mouse-down` and `mouse-up` commands are reserved and return an error for now
Example request
Open System Settings, find the General section and tell me which macOS version I have now, without changing anythingLimitations
- It runs on macOS. The program that launches the command (Terminal, iTerm2, VS Code, Codex) needs the Accessibility permission, and screenshots also need Screen Recording. `agent-desktop permissions --request` asks for what is missing in an isolated helper.
- npm has version 0.9.4, and the author calls the current macOS adapter phase one of the project.
- By default ref actions run without a physical click and without taking focus. Double-click, hover and drag need the `--headed` flag, and then the agent takes the real cursor and focus.
- Installing the package downloads the binary from GitHub releases, so github.com must be reachable. License Apache-2.0.
How to disable. Run `npm uninstall -g agent-desktop` and remove the `agent-desktop` skill folder from the agent's skills directory. Revoke the Accessibility permission in System Settings under Privacy & Security. Session logs live in `~/.agent-desktop/sessions`, and `session gc` clears old ones.
FAQ
Does the agent need screenshots to see the interface?
No. The agent reads the accessibility tree: `snapshot` returns roles, names and refs for buttons, fields and menu items. A screenshot is a separate `screenshot` command and needs the Screen Recording permission.
What does the STALE_REF error mean?
The ref pointed to an element that is gone from the window or has changed. Before acting the program looks the target up again by window, path, role and position and never clicks blindly. Take a new `snapshot` and repeat the action with a fresh ref.
How does skeleton mode help on Slack or VS Code?
A full snapshot of such an app is huge: in the author's Slack example it is 30,743 tokens against 383 for the overview. The `--skeleton` mode shows the tree three levels deep with the number of nested elements. `snapshot --root` expands the region you need.
How do I drive tabs inside an Electron app?
Launch the app with `agent-desktop launch "Obsidian" --cdp`. It opens a debugging port on `127.0.0.1` that Playwright, Puppeteer or `agent-browser` can attach to. Menus and dialogs stay with `agent-desktop`. For an app that is already running, run `close-app` first.
Can I call agent-desktop from something other than the command line?
Yes. The releases include a `libagent_desktop_ffi` library with a C interface: load it from Python, Swift, Go, Ruby, Node or C and call functions without starting a process for every command. The guide is in `skills/agent-desktop-ffi`.
How do several agents work in one session?
`session start --cursor --multi-agent` creates a shared session, and each agent gets its own `AGENT_DESKTOP_AGENT_ID`. Each one shows its own cursor hint with a label. Snapshots stay shared, so act on refs from your own `snapshot`.
Discussion
No questions or reviews yet.
Related
CLI toolsA Swift MCP server that lets an agent drive any macOS app: click, type and read the interface tree
Open Computer Use
open-computer-use
An open MCP server for desktop control: the agent reads app state and clicks, types and scrolls on macOS, Linux and Windows
A plugin and CLI catalog for agents: generates command-line interfaces for GUI apps like GIMP and Blender and installs ready ones via CLI-Hub
Clawd Cursor: desktop control through MCP
Clawd Cursor
An MCP server that lets an agent drive windows and apps: it reads the screen as structure and acts by element name
An MCP server for macOS: the agent moves the mouse, types, manages windows and captures the screen when an app has no API
A self-hosted control plane for running several agents: task dispatch, run review, spend tracking, connected via MCP, CLI or REST
Comparison with similar tools
Entries that share the main search query, and how they differ.
| Entry | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| agent-desktop (this entry) | no | 1,829 | ||
| Open Computer Use | no | not required | 2,756 | an MCP server for desktop control on macOS, Linux and Windows |
| Windows-MCP | no | not required | 10,682 | an MCP server for controlling Windows, works with files and PowerShell |
README in short
The repository holds the core and macOS adapter sources as Rust crates, an npm wrapper, a library with a C interface, three skills with detailed command references, JSON format docs with an FAQ, element lookup accuracy benchmarks and tests. The front page opens with an architecture diagram, followed by installation, permissions, the agent work loop, the full command list and a platform support table.