sim-use CLI
A CLI that shows an agent an iOS Simulator or Android emulator screen as a compact element list and taps elements by short alias.
- 1,541 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The agent gets control of the simulator and device screen: taps, text input, screenshots and video.
- Physical iPhones and iPads attached to a Mac are driven in an experimental mode, so keep test data on them.
Install
- Works in
Manual install
brew install lycorp-jp/tap/sim-useFirst run `brew tap lycorp-jp/tap`. Then `sim-use init --client claude` installs the bundled skill. macOS 14 or newer is required.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The agent gets control of the simulator and device screen: taps, text input, screenshots and video.
- Physical iPhones and iPads attached to a Mac are driven in an experimental mode, so keep test data on them.
This is third-party code. Review the repository files before installing.
What it does
sim-use is a command line for driving a mobile app from an agent loop: look at the screen, tap, verify the result. `sim-use ui` prints the screen as a compact tree of elements numbered like `@9`, and `sim-use tap @9` taps the element without coordinates. The authors say this view is about 16 times shorter than a raw JSON accessibility tree, so a model needs a few hundred tokens for a whole screen. The tool uses Apple accessibility APIs, the iOS Simulator input pipeline and Android AccessibilityService. One command set works with iOS Simulators, Android emulators and devices, while USB-attached iPhones and iPads are supported experimentally with a reduced command set. Besides `ui` and `tap` there are `swipe`, `long-press`, `type`, `paste`, `gesture`, `screenshot`, `record-video` and element selection by `#id` or `--label`. A background daemon runs per device, so one look-and-tap round trip takes about 300 ms. The `sim-use init --client claude` command puts a bundled skill with the full command set into the client's skill directory. Replies can be returned as `--json` with hints on errors.
Who it is for. Mobile developers who want an agent to run the app in a simulator itself and verify that the screen looks and behaves as intended.
Good fit when
- An agent finished a settings screen and should open it in the simulator and tap every item itself.
- A check scenario for iOS and Android should run with one command set.
- You need a video of a scenario run or screenshots after each step.
- An input field takes Cyrillic or emoji and ordinary typing breaks. `paste` covers that.
- You want fewer tokens spent describing the screen in a long verification loop.
Not a fit when
- You need permanent CI test automation, where XCUITest or Maestro with stable scripts fits better.
- The app must run on a real device with the full command set, while only part of the features reach physical iPhones.
- You have no macOS, where some commands, video capture and `init` are macOS-only.
Example request
Launch the app in an iPhone simulator, open the settings screen with sim-use, tap the General item and take a screenshot.Limitations
Building from source needs macOS 14 or newer and a recent Xcode, while the ready version installs through Homebrew. Support for physical iPhones and iPads is experimental. For Android, run `sim-use android init --device <serial>` once, which installs a bridge APK on the device. When the screen tree is empty because another process owns the screen, `ui` tries cross-process discovery and marks the result as flat. Apache-2.0 license.
How to disable. Remove the skill with `sim-use init --uninstall --client claude`, then run `brew uninstall sim-use`. Background daemons end with the session, and the bridge APK on Android you remove by hand.
FAQ
How does the agent pick an element to tap?
By the `@N` alias from the last `ui`, by `#id`, by label with `--label` or by coordinates. The alias is fastest, while the id survives layout changes.
Does it work with Android?
Yes. A bridge APK is installed once with `sim-use android init`, after which the same commands work on an emulator and a device.
What does the `--json` flag give?
A structured reply with a `hint` field on errors, convenient to parse in an agent loop.
Do I need to start the simulator by hand?
It only has to be booted. With one booted simulator the `--device` flag is optional.
How do I connect the skill to Claude Code?
Run `sim-use init --client claude`. With no arguments the command finds installed clients itself.
Discussion
No questions or reviews yet.
Related
CLI toolsThe npx serve-sim command streams an iOS Simulator to the browser and lets an agent tap, swipe, type and replace the camera feed
Baguette CLI saves iOS Simulator screenshots and recordings for review while an agent runs tests without a Simulator.app window.
A CLI that checks SwiftUI screens and runs iOS tests with no Simulator: agents read an accessibility tree and reload views in 2 s.
An agent in an IDE or CLI runs iOS simulators and Android emulators, reads the screen, profiles and runs tests.
A DeepSeek Harness plugin: iOS Simulator and a USB iPhone inside the chat, with 22 tools to drive them
Desktop app and CLI for Android emulators, iOS simulators and real phones with an MCP server for agents
README in short
The authors position sim-use as the missing link in the plan, code, verify, ship loop for mobile development with agents. Written in Swift under Apache-2.0 by LY Corporation.