Midscene Skills
Midscene skills drive a browser, desktop, Android, iOS and HarmonyOS from screenshots and build Vitest E2E tests through Claude Code or OpenClaw
- 321 ★ on the repository
- Last commit
- MIT
Broad access
Why this level
- The agent clicks and types in any window on screen and the authors themselves warn of unpredictable results
- Screen screenshots go to an external model and may carry passwords, messages and personal data
- Through ADB and WebDriverAgent the agent gets control of a phone
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/midscene-skillsDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The agent clicks and types in any window on screen and the authors themselves warn of unpredictable results
- Screen screenshots go to an external model and may carry passwords, messages and personal data
- Through ADB and WebDriverAgent the agent gets control of a phone
This is third-party code. Review the repository files before installing.
What it does
The Midscene.js repository holds seven skills that give an agent screenshot-based UI control. The model looks at the screen, finds the element from the picture and acts, so neither page markup nor accessibility labels are needed. The browser skill runs headless Chrome through Puppeteer and does not take over the user mouse and keyboard. It can attach to an already open Chrome over CDP. The chrome-bridge skill controls the user own browser. For desktops there is computer-automation on macOS, Windows and Linux, and for mobile android-automation through ADB, ios-automation through WebDriverAgent and harmony-automation through HDC. The vitest-midscene-e2e skill scaffolds, converts and maintains E2E tests for web, Android and iOS. Per the browser skill rules the agent runs Midscene commands strictly one at a time in the foreground and reads the screenshot after each command before the next step.
Who it is for. QA engineers and developers who need an AI to check a UI by sight in a browser, on a desktop or on a phone when selectors are missing or brittle.
Good fit when
- The agent just built a page and you need to open it in a browser and confirm the form submits
- A UI without stable selectors, such as a canvas or a third-party widget, needs scenarios driven by the picture
- An Android app is checked on a device through ADB with steps written as plain sentences
- Vitest E2E tests must be set up from scratch or converted from older scenarios
- You want to attach to an open Chrome over CDP and continue a scenario in a ready session
Not a fit when
- You have fast Playwright tests with reliable selectors: screenshot-based work is slower and costs more
- Screenshots of the screen must not be sent to a third-party model cloud
- Mail, a bank or a password manager is open on screen and the AI must not see or touch it
- You have no access to a model with good visual element grounding
Example request
Use the Midscene browser skill to open the login page and check that a wrong password shows an error messageLimitations
- The authors warn in the README that AI-driven UI automation is unpredictable because it can act on anything on screen.
- The skills do not work without a separately configured model with strong visual grounding: key, name, address and family go into `MIDSCENE_MODEL_API_KEY`, `MIDSCENE_MODEL_NAME`, `MIDSCENE_MODEL_BASE_URL` and `MIDSCENE_MODEL_FAMILY`.
- Node.js is required, and phones also need ADB, WebDriverAgent or HDC.
- The repository takes bug reports only in the main Midscene project.
How to disable. Delete the Midscene skill folders from the agent skills directory. Remove the `MIDSCENE_MODEL_*` variables and the model key from the environment and from the `.env` file.
FAQ
How do I install Midscene Skills in Claude Code?
Run `npx skills add web-infra-dev/midscene-skills -a claude-code`. For OpenClaw set the flag value to `openclaw`. Node.js must be installed.
Which models fit?
The README names Doubao Seed 1.6, Qwen3-VL, Zhipu GLM-4.6V, Gemini-3-Pro and Gemini-3-Flash. You need a model with good visual grounding, and access is set through the `MIDSCENE_MODEL_*` variables.
Will the browser skill take over my mouse?
No, it runs in headless Puppeteer and leaves your cursor alone. For your own browser use the chrome-bridge skill, and computer-automation drives the whole desktop.
How do I control a phone?
Android connects through ADB, iOS through WebDriverAgent, HarmonyOS through HDC. Each platform has its own skill.
Can the skill be used for scraping?
Yes, the browser skill lists collecting data from websites among its tasks. Site terms and personal data law remain your responsibility.
Discussion
No questions or reviews yet.
Related
SkillsA skill that tests your agent's code in a real browser: reads your changes, builds a test plan and runs it through Playwright with your own cookies
Runs Maestro flows on Android, iOS and in the browser as a single binary without a JVM and installs a skill for the agent.
A skill that lets an agent write and run Playwright scripts: page tests, forms, screenshots and link checks
A skill giving Claude Code a background browser: screenshots at three sizes, frontend audits, before and after diffs, login flows.
One binary for agent browser automation: a CLI with browser and check skills, an MCP server and clients for JS, Python and Java
Explorbot autonomously explores a web app in the browser, finds bugs and saves flows as runnable Playwright tests.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| Midscene Skills (this card) | yes | 321 | ||
| Playwright Skill | no | 3,287 | Playwright scenarios with selectors, while Midscene relies on screenshots and needs no markup | |
| agent-browser | yes | not required | 50,530 | a browser control CLI built on the accessibility tree, without image-based recognition |
README in short
The page opens with a list of platforms and skills with their `skills/` paths, then a risk warning. After it come install commands, four model setup examples and two sample prompts for the agent. It ends with a section on reporting issues and the MIT license.
README badge
Original SKILL.md
---
name: browser-automation
description: |
Vision-driven browser automation using Midscene. Operates from screenshots — no DOM or accessibility labels needed.
Runs in headless Puppeteer — does NOT take over the user's mouse or keyboard.
Also supports CDP mode and Bridge mode to connect to an existing Chrome.
Use this skill when the user wants to:
- Browse, navigate, or open web pages
- Scrape, extract, or collect data from websites
- Fill out forms, upload local files, click buttons, or interact with web elements
- Verify, validate, test, or QA frontend UI behavior
- Take screenshots of web pages
- Automate multi-step web workflows
- Test what was just built, see if it works in browser
- Connect to Chrome via CDP, DevTools Protocol, or remote debugging
- Connect to user's Chrome browser, control my browser, operate my Chrome
Powered by Midscene.js (https://midscenejs.com)
allowed-tools:
- Bash
---
# Browser Automation
> **CRITICAL RULES — VIOLATIONS WILL BREAK THE WORKFLOW:**
>
> 1. **Never run midscene commands in the background.** Each command must run synchronously so you can read its output (especially screenshots) before deciding the next action. Background execution breaks the screenshot-analyze-act loop.
> 2. **Run only one midscene command at a time.** Wait for the previous command to finish, read the screenshot, then decide the next action. Never chain multiple commands together.
> 3. **Allow enough time for each command to complete.** Midscene commands involve AI inference and screen interaction, which can take longer than typical shell commands. A typical command needs about 1 minute; complex `act` commands may need even longer.
> 4. **Always report task results before finishing.** After completing the automation task, you MUST proactively summarize the results to the user — including key data found, actions completed, screenshots taken, and any relevant findings. Never silently end after the last automation step; the user expects a complete response in a single interaction.
Automate web browsing using `npx -y @midscene/web@1`. By default, launches a headless Chrome via Puppeteer that **persists across CLI calls** — no session loss between commands. Also supports **CDP mode** and **Bridge mode** to connect to an existing Chrome browser.
## What `act` Can Do
Inside a single `act` call in the browser, Midscene can click, right-click, double-click, hover, type or clear text, press keys, scroll, drag, long-press, and continue through multi-step page flows based on what is currently visible. When touch input is enabled, it can also handle swipe- or pinch-style interactions on touch-oriented pages.
## When to Use
This skill has three modes. Choose based on the user's intent:
### Mode Selection Guide
| Mode | When to use | How it works |
|------|------------|-------------|
| **Puppeteer (default)** | User wants to browse a URL, scrape data, test UI — no need for their own browser | Launches a new headless Chrome, isolated from user's browser |
| **CDP mode** | User says "connect to my Chrome", "control my browser", "CDP", "remote debugging", or wants to operate their existing browser. Also use when the task **implicitly requires login state** (e.g., "check my orders", "open my dashboard", "look at my account") | Connects to user's Chrome via DevTools Protocol. Requires remote debugging enabled (`chrome://inspect` > "Allow remote debugging"). No extension needed |
| **Bridge mode** | User explicitly mentions "bridge", "extension", or has Midscene Chrome Extension installed and prefers to use it | Connects to user's Chrome via the Midscene Chrome Extension |
**CDP vs Bridge**: Both control the user's real Chrome with login sessions preserved. CDP only needs a Chrome setting toggle; Bridge needs a Chrome Extension installed. If the user doesn't specify, prefer **CDP mode** as it has fewer prerequisites.
### Precheck: detect available CDP target
Before using CDP mode, run a quick precheck to verify Chrome's remote debugging port is reachable. This avoids long timeouts when the user hasn't enabled remote debugging.
```bash
# CDP precheck (port 9222, 2s timeout) — returns "101" if Chrome is listening for DevTools
curl -s --max-time 2 -o /dev/null -w "%{http_code}" -H "Upgrade: websocket" -H "Connection: Upgrade" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" http://127.0.0.1:9222/devtools/browser