pagesift MCP
A self-hosted Rust service for agents: pages, PDFs and Office files to Markdown, Firecrawl-style API, MCP included
- 244 ★ on the repository
- Last commit
- AGPL-3.0
Extended access
Why this level
- The agent directs the server to fetch any address and launch a windowless browser, and incoming HTTP access to the shared service relies on a key alone.
Install
- Works in
Manual install
cargo install pagesift --lockedInstalls the command-line tool from crates.io. For Python there is `pip install pagesift`, for Node.js `npm install @pagesift/node`.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The agent directs the server to fetch any address and launch a windowless browser, and incoming HTTP access to the shared service relies on a key alone.
This is third-party code. Review the repository files before installing.
What it does
pagesift fetches a page by URL (a browser starts only where the content is invisible without one) or reads a local file, keeps the main text and returns Markdown with metadata. A single Rust binary works as a command, a Python and Node.js library, an HTTP server and an MCP server. Input covers HTML, PDF, Word, Excel, PowerPoint, EPUB and CSV. The `crawl` command walks a site respecting `robots.txt` and pauses between requests to one host, supports a query-focused crawl, and `map` lists a site's addresses. The HTTP server repeats the Firecrawl v1 and v2 API and a `/r/<url>` address in the spirit of Jina Reader, so client code only needs to point at your own server. The MCP server starts with `pagesift mcp` over stdio, or with `pagesift mcp http` as a shared local service. The agent gets the tools `pagesift_scrape`, `pagesift_batch_scrape`, `pagesift_crawl`, `pagesift_crawl_status` and `pagesift_map`.
Who it is for. Agent and RAG developers who need their own text extraction service instead of paid cloud APIs.
Good fit when
- An agent reads documentation and articles and needs clean Markdown with headings, lists and tables.
- Cloud Firecrawl or Jina Reader is replaced by your own server: only the address changes in client code.
- A mix of HTML, PDF, Word, Excel and e-books is loaded into a RAG index in one call.
- A site is crawled politely: respecting `robots.txt`, per-host limits and backoff on 429 and 503 errors.
- The service runs in a small container where memory is bounded and sized from the container limit.
Not a fit when
- Scanned PDFs and images: pagesift reads only the text layer and does not recognize text.
- You want the fastest possible crawl with no pauses: an interval is kept between requests to one host, and `spider` is clearly faster on raw speed.
- The AGPL-3.0 license does not fit and a commercial one is out: pick a permissively licensed tool.
- You need a managed cloud service: pagesift has no hosted API yet.
Example request
Use pagesift to crawl the documentation section `https://docs.rs/tokio` up to 50 pages and save the Markdown to the `./data/` folder.MCP
- Transport
- stdio, http
- Authentication
- not required
Limitations
The AGPL-3.0 license obliges you to open the source of derived work. For closed embedding or private deployment without those obligations a commercial license exists. You install and run the service yourself; there is no cloud. Guards against requests into the internal network and DNS rebinding are on by default, and the needed subnets are opened with the `--allow-cidr` flag. The shared MCP service over HTTP listens on 127.0.0.1 only. In a local install the `pagesift doctor` command shows which browsers and components were found, while the Docker image already ships a browser.
How to disable. Remove the MCP entry with `claude mcp remove pagesift`, stop the container or the `pagesift mcp http` service and delete the binary with `cargo uninstall pagesift`.
FAQ
How do I connect pagesift to Claude Code?
Install the binary with `cargo install pagesift --locked`, then run `claude mcp add pagesift -- pagesift mcp`. The server starts over stdio and gives the agent scraping tools.
Which tools does the pagesift MCP server expose?
Five: `pagesift_scrape` for one page, `pagesift_batch_scrape` for a list of addresses, `pagesift_crawl` for a site crawl, `pagesift_crawl_status` for checking a crawl job and `pagesift_map` for a list of addresses.
Does pagesift replace Firecrawl and Jina Reader?
On the API side, yes: the server repeats the `scrape`, `batch/scrape`, `crawl` and `map` endpoints of v1 and v2 and the `/r/<url>` address. Code with the Firecrawl client is switched by setting `api_url` to your server, for example `http://localhost:3002`.
How do I give one MCP service to several agents?
Start `pagesift mcp http --port 3003` and connect the agents with `claude mcp add --scope user --transport http pagesift http://127.0.0.1:3003/mcp`. Browsers and cache are shared across agents and every request lands in an audit log.
How do I check that the environment is ready?
Run `pagesift doctor`: the command checks browsers and the environment. The `ghcr.io/yolow92/pagesift` image ships with a browser, so a container needs no extra setup.
Discussion
No questions or reviews yet.
Related
MCP serversA Rust server that turns pages into Markdown or JSON: search, scrape, site map, crawl and PDF through MCP, a CLI and a Firecrawl-compatible API
A plugin that crawls sites and turns pages into Markdown with metadata: seven skills and an MCP server with nine tools
MCP server that reads sites on your machine and hands the agent short Markdown with links, keeping raw HTML out of context.
An open-source crawler that turns websites into clean Markdown for AI agents: install from PyPI, run in Docker and connect to Claude Code over MCP
Official Firecrawl MCP server: web search, page scraping to Markdown, site crawling and in-page actions
The Firecrawl command-line client with agent skills: web search, scraping, crawling, URL mapping and page interaction
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| pagesift (this card) | no | not required | 244 | |
| Crawl4AI | yes | API key | 86,959 | a Python crawler; pagesift is a single Rust binary with Firecrawl and Jina Reader compatible APIs |
| Firecrawl MCP Server | yes | API key | 7,765 | the official MCP for cloud Firecrawl; pagesift offers a similar API on your own server with no per-page fees |
| MarkItDown | yes | not required | 194,189 | converts files of many formats to Markdown; pagesift adds site crawling, an HTTP server and an HTTP MCP mode |
README in short
The README opens with an install table for the CLI, Python, Node.js, the server and MCP, then compares pagesift with other tools. It gives speed and memory measurements on open datasets and separately lists the tasks where pagesift falls behind.