Anansi MCP
A scraper for hostile sites that repairs broken selectors on its own and starts a browser only when a page is empty without one.
- 117 ★ on the repository
- Last commit
- Apache-2.0
Broad access
Why this level
- The tool matches TLS fingerprints and browser headers to pass bot protection: a site may block your IP, and collecting against its terms breaches the contract and the law.
- An agent starts background crawls across hundreds of pages, and the `proxy` parameter takes a proxy login and password directly in the call.
Install
- Works in
Manual install
pip install "git+https://github.com/mdowis/anansi"The first README Install command installs the core. For browser fetching also run `playwright install chromium`.
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The tool matches TLS fingerprints and browser headers to pass bot protection: a site may block your IP, and collecting against its terms breaches the contract and the law.
- An agent starts background crawls across hundreds of pages, and the `proxy` parameter takes a proxy login and password directly in the call.
This is third-party code. Review the repository files before installing.
What it does
Anansi is a Python scraper built to survive layout changes. CSS selectors carry confidence scores: when one breaks, four repair strategies compete and the winner is saved. A page that is only a JS shell is re-requested in a hidden Playwright browser, and the result is cached per domain. The library ships with the `anansi-mcp` server on FastMCP, which exposes 17 tools including `fetch_url`, `fetch_and_extract`, `crawl_site`, `export_crawl` and `screenshot_url`. An agent starts a crawl as a background job, reads items in pages and exports them to JSONL, JSON or CSV. On a re-crawl, unchanged pages are skipped by ETag, Last-Modified, content hash and sitemap `lastmod`.
Who it is for. Scraper developers and analysts who collect data from dynamic sites and want to drive crawls from Claude Code or Cursor.
Good fit when
- A target site changes its layout every few weeks and the parser keeps breaking.
- A page serves an empty JS shell and the data appears only after rendering.
- An agent crawls a catalog section of hundreds of pages as a background job and exports the result to CSV.
- Repeated crawls of one site waste time on pages that have not changed.
- A team checks how its own site reacts to automated requests.
Not a fit when
- The site terms or `robots.txt` forbid automated collection.
- You need a one-off download of a single page that a plain HTTP request covers.
- Access to the data is restricted and you have no right to it.
- You need a hosted service with support and uptime guarantees.
Example request
Crawl the catalog section on shop.example.com, extract the title, price and SKU of each product and export to CSV.MCP
- Transport
- stdio, sse
- Authentication
- not required
Environment variables
ANANSI_IMPERSONATE- curl-cffi TLS and HTTP/2 fingerprint target, for example chrome124.
ANANSI_BROWSER_ENGINE- The value patchright enables the optional stealth browser engine.
Limitations
The base install comes straight from the GitHub repository with `pip`, and browser mode needs a separate `playwright install chromium`. TLS fingerprint mimicry and the Patchright engine come as extra packages. The user is responsible for lawful collection: follow site terms, `robots.txt`, computer-misuse law and data-protection law. The anti-bot features are meant for authorized testing and for content you have the right to access. On Windows a client may fail to find `anansi-mcp`; run the server as `python -m anansi.mcp_server.server` then.
How to disable. Remove the anansi entry from the MCP client configuration and uninstall the package with `pip uninstall anansi-scraper`. Learned selectors and crawl results persist on disk; clean them separately.
FAQ
How do I connect Anansi to Claude Code?
Run `claude mcp add anansi -- anansi-mcp`. For Cursor, Windsurf and Claude Desktop, put the `anansi-mcp` command in the client configuration. If Windows cannot find it, use `python -m anansi.mcp_server.server`.
Which tools does the MCP server have?
There are 17 in total. `fetch_url` and `fetch_and_extract` read pages, `crawl_site`, `pause_crawl`, `resume_crawl` and `export_crawl` manage crawls, and `selector_health`, `train_selector` and `validate_selector` check selectors.
How do I get a long page in parts?
Set the `fetch_url` format to `text` or `markdown`, or pass `chunk_size` and `chunk_index`. The page stays cached for 5 minutes, so later chunks arrive without a new request.
Can Anansi click buttons on a page?
Yes. With `use_browser=true`, the `actions` list runs `click`, `fill`, `press`, `scroll_until_stable` and waits. `capture_network` returns the JSON API responses the page requests while loading.
Can the anti-bot features be used on any site?
No. TLS fingerprints, personas and Cloudflare, Akamai and DataDome handling are meant for authorized testing and for content you have the right to access. You follow site terms and `robots.txt` yourself.
Discussion
No questions or reviews yet.
Related
MCP serversAn MCP server and CLI that crawls sites: a small model picks links and passages so the agent reads only the answer.
A plugin that crawls sites and turns pages into Markdown with metadata: seven skills and an MCP server with nine tools
A web research skill: SearXNG searches, Camofox reads pages and CloakBrowser opens sites that check their visitors
MCP server with DuckDuckGo search and page text extraction: several URLs can be passed in one call
Local web search for agents: searches, reads pages and returns compact passages with sources, saving context tokens
The Firecrawl command-line client with agent skills: web search, scraping, crawling, URL mapping and page interaction
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| Anansi (this card) | no | not required | 117 | |
| Crawl4AI | yes | API key | 87,271 | Crawl4AI turns sites into Markdown for agents; Anansi remembers repaired selectors and runs re-crawls that skip unchanged pages |
| Firecrawl MCP Server | yes | API key | 7,809 | Firecrawl MCP serves pages through the hosted Firecrawl API; Anansi installs locally and repairs selectors itself |
README in short
A library for Python 3.11 and newer with a ready MCP server. The `docs/` folder covers installation, capabilities, internals, the identity layer and client setup.