Crawl4AI MCP
An open-source crawler that turns websites into clean Markdown for AI agents: install from PyPI, run in Docker and connect to Claude Code over MCP
- 86,076 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The server visits any address from a managed browser, runs JavaScript and takes page screenshots on its own
- The cloud MCP sends addresses and page content to the developer's servers and spends credits on your key
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add mcp/crawl4aiDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The server visits any address from a managed browser, runs JavaScript and takes page screenshots on its own
- The cloud MCP sends addresses and page content to the developer's servers and spends credits on your key
This is third-party code. Review the repository files before installing.
What it does
Crawl4AI loads pages in a managed browser and returns them as clean Markdown with headings, tables, code blocks and numbered citations. Filters such as BM25ContentFilter strip menus, footers and boilerplate and leave a fit Markdown version for your query. Data can be extracted in a structured way: CSS and XPath schemas with no model, LLMExtractionStrategy into typed JSON, or the generate_schema helper. Deep crawling supports BFS, DFS and best-first strategies, and AdaptiveCrawler stops once it has gathered enough to answer your query. The browser can use a persistent profile with saved logins, proxies, sessions and a stealth mode. There are three ways to run it: the Python library, the crwl console command and a Docker server with REST endpoints /md, /crawl, /screenshot, /pdf and /execute_js. That server exposes MCP at /mcp/sse and /mcp/ws with the tools md, html, screenshot, pdf, execute_js, crawl and ask. The developer also offers a hosted Crawl4AI Cloud with an MCP address at api.crawl4ai.com/mcp.
Who it is for. For developers of RAG pipelines and agents who regularly turn websites and documentation into Markdown and structured data without building their own browser infrastructure.
Good fit when
- You need a site's docs as Markdown for an index: crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 walks the pages breadth first
- Pages that load dynamically must be scrolled and awaited: the scan_full_page parameter and selector waits handle infinite feeds
- You must get JSON from dozens of similar product cards with no model cost, and JsonCssExtractionStrategy schemas fit
- An agent in Claude Code should take screenshots, PDFs and Markdown of pages itself through a local Docker server over MCP
- You need to sign in under your own account with saved cookies: a persistent browser profile keeps the session between runs
Not a fit when
- You need a one-off page fetch inside an agent: the built-in WebFetch and search in Claude Code are enough with no browser or Docker
- A site's terms forbid automated collection: stealth mode and proxies leave that restriction in force, and you carry responsibility for it
- Docker is missing and you cannot install a Python package with Playwright browsers, while the hosted service does not suit you because it is paid in dollars
- You need structured search engine results: the /search and /serp endpoints exist only in the paid cloud
Example request
Crawl the docs section at the linked address two levels deep, save every page as Markdown and build a table of contentsMCP
- Transport
- sse, http
- Authentication
- API key
Environment variables
CRAWL4AI_API_TOKENrequired, secret- Token of your own Docker server; endpoints are closed without it
CRAWL4AI_KEYsecret- Crawl4AI Cloud key, needed only for the cloud MCP
Limitations
- The library installs with pip install -U crawl4ai, then crawl4ai-setup downloads the browser. crawl4ai-doctor checks the install.
- A Docker server without the CRAWL4AI_API_TOKEN token answers only inside its container. Allow about ten seconds after start.
- LLM extraction and page questions in crwl need a key for the chosen model, set with crwl config.
- The /search, /serp and /answer endpoints exist only in the cloud. SERP is marked beta and answer experimental.
- The cloud is paid: after email verification you get 1,000 free credits a month, and plans start at 10 dollars a month.
- The ask MCP tool answers questions about the Crawl4AI library itself and says nothing about a loaded page.
How to disable. Remove the server from Claude Code with claude mcp remove c4ai-sse (or crawl4ai for the cloud). Stop the Docker container with docker stop crawl4ai, delete it with docker rm crawl4ai and uninstall the library with pip uninstall crawl4ai.
FAQ
How do I install Crawl4AI and run a first crawl?
Run pip install -U crawl4ai, then crawl4ai-setup to download the browser. The README's Python example is AsyncWebCrawler().arun(url=...), and the output is in result.markdown. From the console, crwl https://news.ycombinator.com -o markdown gives the same result.
How do I connect Crawl4AI to Claude Code over MCP?
For a local Docker server run claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse. The cloud uses a different command: claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp --header "Authorization: Bearer $CRAWL4AI_KEY". Instructions for Codex, Cursor and OpenCode are in the developer's docs.
Which MCP tools does the local Crawl4AI server provide?
Seven: md, html, screenshot, pdf, execute_js, crawl and ask. Detailed parameter schemas are served at http://localhost:11235/mcp/schema.
How does the cloud differ from running Crawl4AI yourself?
In the cloud the developer runs the browsers and proxies, JS-heavy pages and bot walls are handled automatically, and the key also gives /search, /serp and /answer. Self-hosting means you tune browsers and proxies yourself but pay for no credits.
How do I get structured data without a model?
Describe a CSS or XPath schema and pass it to JsonCssExtractionStrategy or JsonXPathExtractionStrategy. If writing the schema is tedious, generate_schema drafts one from a task description with the help of a model, and the schema can then be reused.
Discussion
No questions or reviews yet.
Related
MCP serversA Rust server that turns pages into Markdown or JSON: search, scrape, site map, crawl and PDF through MCP, a CLI and a Firecrawl-compatible API
The Firecrawl command-line client with agent skills: web search, scraping, crawling, URL mapping and page interaction
Official Firecrawl MCP server: web search, page scraping to Markdown, site crawling and in-page actions
An MCP server that opens pages in headless Chromium through Playwright and hands the agent the main text as Markdown, even from JavaScript-heavy sites
Local-first Rust tool for web extraction: scrape, crawl and structured data for LLMs, with a CLI, REST API and MCP server
Official Oxylabs MCP server: page scraping with JS rendering, Google and Amazon results, an AI crawler
Comparison with similar tools
Entries that share the main search query, and how they differ.
| Entry | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| Crawl4AI (this entry) | yes | API key | 86,076 | |
| Firecrawl MCP Server | yes | API key | 7,672 | the official MCP of the hosted Firecrawl service, while Crawl4AI can be fully self-hosted |
| fastCRW | yes | not required | 1,172 | a Rust server that turns pages into Markdown or JSON |
README in short
The page begins with a choice between running it yourself and the cloud, with examples for both and a comparison table of the library, a Docker server and the cloud. Collapsible blocks then cover the features: Markdown, data extraction, browser control, crawling, self-hosting and what the cloud adds. Installation through pip, Docker and the command line follows, then advanced code examples, sponsors and documentation links.