doc-scraper MCP
A Go crawler turns documentation sites into Markdown and gives agents offline search_docs and page reading over MCP.
- 109 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The tool sends requests to third-party sites, and mass crawling loads other people servers.
- Downloaded text reaches the agent context as is, so a page with hidden instructions can influence answers.
Install
- Works in
Manual install
go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latestThe first README command. Binary releases, a Docker image and the `doc-scraper.mcpb` package also exist.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The tool sends requests to third-party sites, and mass crawling loads other people servers.
- Downloaded text reaches the agent context as is, so a page with hidden instructions can influence answers.
This is third-party code. Review the repository files before installing.
What it does
`doc-scraper` crawls technical documentation sites, extracts the main content and saves pages as Markdown with frontmatter, plus `pages.jsonl`, `llms.txt` and `llms-full.txt`. It recognizes more than 30 documentation generators, respects `robots.txt` and sitemaps, rate-limits requests, resumes interrupted crawls and refreshes only what changed (`get_freshness`, `diff_crawl`). The MCP server starts with `doc-scraper mcp-server -config ...` and gives the agent 13 tools, among them BM25 search `search_docs` and `read_page` over the stored corpus without network. The agent works with the documentation you chose rather than model memory.
Who it is for. Developers whose agents mix up library versions and who need a pinned offline documentation corpus.
Good fit when
- An agent gets a fresh library API wrong and its documentation is needed at hand.
- Documentation is read offline: on the road or inside a closed network.
- A corpus is refreshed on a schedule to see what changed since the last crawl.
- A documentation site is turned into `llms-full.txt` for another tool.
Not a fit when
- You need to crawl arbitrary sites behind bot protection; the tool respects `robots.txt` and rate limits.
- Pages are built entirely by JavaScript; the tool flags such shells instead of crawling them.
- You need a single page once; an ordinary page read is enough.
Example request
Crawl the Rust CLI book documentation, find the error-handling section and show how it is written.MCP
- Transport
- stdio
- Authentication
- not required
Limitations
You need to install the binary (release, `go install` or Docker) and configure a site with `doc-scraper add`. Sites rendered entirely by JavaScript are reported by the crawl but not read. The corpus lives in `./crawled_docs/<site>/`, and data freshness depends on repeated crawls. MCP runs over `stdio`, and Claude Desktop has the `doc-scraper.mcpb` package. There is no telemetry: network requests go only to the sites you specify.
How to disable. Remove the MCP server from the client configuration and delete the binary. Delete the `crawled_docs` directory and crawl state separately if the corpus is no longer needed.
FAQ
How do I try the tool quickly?
Run `doc-scraper add -site rust_cli_book <url>`, then `doc-scraper crawl -site rust_cli_book`. Pages appear in `./crawled_docs/rust_cli_book/`.
How do I connect it to Claude Code?
With `claude mcp add --transport stdio doc-scraper -- /path/to/doc-scraper mcp-server -config /path/to/config.yaml`.
Does the agent use the network when searching?
No. `search_docs` and `read_page` work over the stored corpus offline. The network is needed only for the crawls.
Is there a package for Claude Desktop?
Yes, the `doc-scraper.mcpb` file from the latest release installs instead of manual setup.
How many tools does the MCP server have?
Thirteen, including `search_docs`, `read_page`, `get_freshness` and `diff_crawl`. The full list is in the project documentation.
Discussion
No questions or reviews yet.
Related
MCP serversRef's MCP server: documentation search for APIs and libraries that hands the agent only the relevant parts of pages
Docs MCP Server
Grounded Docs MCP Server
A local MCP server that indexes docs from websites, GitHub, npm, PyPI and files and gives the agent version-aware search
MCP server that fetches the exact documentation section instead of the whole file: a heading index and byte-precise reads
Deepwiki MCP
Deepwiki MCP Server
MCP server that fetches deepwiki.com pages, cleans them and returns Markdown to the agent, as one document or split by page
A remote MCP server that gives the agent docs and code of any public GitHub repository with nothing to install
An MCP server that graphs repository dependencies, writes docs from the graph and answers agent questions via codebase-qna
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| doc-scraper (this card) | no | not required | 109 | |
| Docs MCP Server | no | not required | 1,875 | indexes documentation for MCP search; doc-scraper crawls sites itself and keeps a Markdown corpus with change history |
| Crawl4AI | yes | API key | 87,271 | a general crawler for data; here crawling is tuned for documentation sites and offline agent search |
README in short
Crawling is polite and resumable: staged output never overwrites the last good copy. Apache-2.0 license, documentation on a separate site and a generator-detection benchmark.