DesktopCtl CLI
A local macOS CLI that lets an agent read the screen via OCR, click and type, with no cloud and no vision model
- 71 ★ on the repository
- Last commit
- MIT
Broad access
Why this level
- The agent gets mouse and keyboard control and can read the whole screen, including open apps such as mail and messengers
- Accessibility and Screen Recording permissions are required, and the install is a script from the internet with a local signature and no notarization
Install
- Works in
Manual install
curl -fsSL https://desktopctl.com/install.sh | bashmacOS on Apple Silicon only. A script from the internet, so read it before running. Accessibility and Screen Recording permissions are needed.
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The agent gets mouse and keyboard control and can read the whole screen, including open apps such as mail and messengers
- Accessibility and Screen Recording permissions are required, and the install is a script from the internet with a local signature and no notarization
This is third-party code. Review the repository files before installing.
What it does
DesktopCtl lets any agent control a computer through the screen, mouse and keyboard. It has three parts: the DesktopCtl.app menu bar app with a launcher and settings, the desktopctld service that owns perception, state, execution and verification, and a stateless desktopctl CLI that sends commands and queries over local IPC. Everything runs locally, no screenshots go to the cloud, and on-screen text is recognized with GPU acceleration. The desktopctl screen tokenize command turns a window into a structured set of tokens: OCR text and interface elements with predictable ids such as button_7 or text_settings. That is why any model works, even one without vision. Actions are addressed by selector (--text, --token) with a coordinate fallback. The app open, window focus, pointer click, keyboard type and keyboard press commands work on a window by its id, while menu list and menu click invoke menu items through Accessibility. After each action a loop watches the screen for changes by default, and --no-observe turns it off. There are explicit waits and JSON responses with a request_id and stable error codes. The repository also carries two skills: one for Obsidian (through its CLI) and one for reminders.
Who it is for. macOS users on Apple Silicon who want to give Claude Code, Codex or another agent desktop control and keep screen data on their own machine.
Good fit when
- An agent opens an app, finds a window by id and types into the right field
- A model without vision is used and works from tokens and the screen text produced by OCR
- Screen captures must not go to the cloud, and recognition runs on the machine itself
- An agent invokes a menu item by the id menu_file_new_window instead of hunting with the mouse
- Automation waits for text to appear or disappear on screen and verifies the final window state
Not a fit when
- Your machine runs Windows or Linux: only macOS on Apple Silicon is supported
- You need automation without granting permissions: the app needs Accessibility and Screen Recording access
- The agent controls an app that has an API or a command line: the direct path is more reliable than clicking the screen
- You need a production-grade signature: the build is signed locally and not notarized with a Developer ID
Example request
Open the Notes app through desktopctl, find the window, type "Shopping list" and verify with screen tokenize that the text appearedLimitations
- Only macOS on Apple Silicon is supported, and other platforms are listed as plans.
- The app needs Accessibility and Screen Recording permissions.
- The build is signed locally and not notarized with a Developer ID. The installer removes the com.apple.quarantine attribute only from the installed app and CLI and leaves system Gatekeeper settings alone.
- The service runs one command at a time, and a queue wait of more than about 5 seconds ends with a TIMEOUT error and exit code 3.
- The author calls the project active development: planned are a doctor command, failure explanations and better local vision.
- The license is MIT.
How to disable. DesktopCtl.app is deleted from ~/Applications, the CLI from ~/.local/bin and the data from ~/.local/share/desktopctl or the DESKTOPCTL_HOME folder. The granted Accessibility and Screen Recording permissions are revoked in macOS system settings.
FAQ
How do I install DesktopCtl?
With curl -fsSL https://desktopctl.com/install.sh | bash. The installer puts the CLI at ~/.local/bin/desktopctl and the app at ~/Applications/DesktopCtl.app. It works only on macOS with Apple Silicon.
Do screenshots go to the cloud?
No, the tool runs locally: text recognition happens on the machine with GPU acceleration and there are no cloud dependencies.
Do I need a vision model?
No. The screen tokenize command returns screen text and interface elements with ids, so any model works, including one without image support.
Which permissions does the app need?
Accessibility and Screen Recording for DesktopCtl.app. They are granted in macOS system settings.
How do I invoke a menu item?
Run desktopctl menu list to get ids such as menu_file_new_window, then desktopctl menu click --id <id>. An exact flattened title path works too.
Discussion
No questions or reviews yet.
Related
CLI toolsA Rust CLI for macOS: an agent reads any app's interface through the accessibility tree and clicks, types and scrolls by element refs, with no screenshots
An MCP server that lets an agent see the screen and control macOS, Windows and Linux: screenshots, clicks, OCR and screen memory
Open Computer Use
open-computer-use
An open MCP server for desktop control: the agent reads app state and clicks, types and scrolls on macOS, Linux and Windows
A Swift MCP server that lets an agent drive any macOS app: click, type and read the interface tree
A macOS CLI and MCP server: screenshots, accessibility-based UI inspection and app control via clicks and typing
Mano skill
Mano skill (mano-cua)
A CLI and skill for desktop control from a text task: screenshots, clicks, typing and scrolling, in the cloud or locally on Apple Silicon
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| DesktopCtl (this card) | no | 71 | ||
| agent-desktop | no | 22,602 | another desktop control tool for agents with its own command set | |
| mcp-screenshot-website-fast | no | not required | 111 | takes website screenshots for an agent and does not control windows or the mouse |
README in short
The page presents DesktopCtl as a local CLI for AI agents and lists its strengths and three architecture components. Next come the file and directory layout, the current scope (macOS, OCR, tokens, menus), requirements and one-command install. The quick start shows an example with Notes and a menu call, and status and plans close the page.