Crawlberg plugin
A plugin that crawls sites and turns pages into Markdown with metadata: seven skills and an MCP server with nine tools
- 195 ★ on the repository
- Last commit
- MIT
Broad access
Why this level
- The agent visits external sites and can drive a browser, click and run scripts on pages
- The MCP launcher downloads and runs a binary through npx, Homebrew or a prebuilt build
- Bot-filter bypass is advertised: use it only on sites where collection is permitted
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add plugins/crawlbergDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The agent visits external sites and can drive a browser, click and run scripts on pages
- The MCP launcher downloads and runs a binary through npx, Homebrew or a prebuilt build
- Bot-filter bypass is advertised: use it only on sites where collection is permitted
This is third-party code. Review the repository files before installing.
What it does
The plugin plugs the Rust-based Crawlberg engine into the agent. It fetches one page or a whole site, cleans HTML into Markdown and returns structured data: headings, links, images, JSON-LD, Open Graph and hreflang. The skills tell the agent which subcommand to pick: `scrape` for a single page, `crawl` for link-following with depth and page limits, `map` for a URL list from links and sitemaps. The MCP server starts with `crawlberg mcp` and exposes nine tools: `scrape`, `crawl`, `map`, `batch_scrape`, `batch_crawl`, `interact`, `download`, `generate_citations` and `get_version`. For JavaScript pages the engine falls back to a headless browser. The `interact` tool runs clicks, typing, scrolling and scripts in the browser before the page is captured. Requests to internal and private addresses are blocked by default, and robots.txt and per-domain rate limits are honored.
Who it is for. RAG developers and analysts whose agent has to collect site content as Markdown without a custom parser.
Good fit when
- Product documentation has to be exported whole to Markdown for an index or the agent context
- You need every URL of a site from links and sitemaps to plan a crawl or a migration
- A page renders with JavaScript and a plain request returns an empty shell
- A page needs preparation before capture: open a menu, scroll a feed, take a screenshot
- Hundreds of URLs are processed in a batch with concurrency and rate limits
Not a fit when
- Scraping closed sections the site owner has not allowed: site rules and the law stay your responsibility
- Proxy pools, long-lived authenticated sessions and scheduling: they live in the separate commercial xberg-enterprise product
- Alpine with musl: some language bindings ship no prebuilt artifacts, while the CLI and Docker image work
- A single page in a browser where copying the text by hand is simpler
Example request
Use crawlberg: crawl docs.example.com three levels deep, no more than 200 pages, and save each page as Markdown into the `docs-md` folderLimitations
- The plugin calls the local `crawlberg` binary. The MCP launcher looks for it in `PATH`, then tries `npx`, Homebrew or a prebuilt download, so the first run reaches the network to install it.
- Headless mode needs Chrome or Chromium installed.
- The `interact` tool can change page and application state: clicks and typing are real.
- The README states that bot filters are detected and worked around. Whether that is lawful depends on the site and its access terms.
How to disable. Remove the plugin with `/plugin uninstall crawlberg@crawlberg`. The `crawlberg` binary fetched by the launcher is removed separately: `brew uninstall crawlberg` if it came from Homebrew.
FAQ
How do I install Crawlberg in Claude Code?
Run `/plugin marketplace add xberg-io/crawlberg`, then `/plugin install crawlberg@crawlberg`. The MCP server fetches the binary on first launch, so no separate install is needed.
Which tools does the Crawlberg MCP server expose?
Nine: `scrape`, `crawl`, `map`, `batch_scrape`, `batch_crawl`, `interact`, `download`, `generate_citations` and `get_version`.
How is the plugin different from the Crawlberg library?
The library is imported from code in 14 languages. The plugin gives the agent ready skills and an MCP server over the local binary, so no code is needed.
What about a page that renders with JavaScript?
`--browser-mode auto` or `always` switches on the headless browser. Chrome or Chromium is required. `never` turns it off and speeds things up when pages are static.
Does Crawlberg reach internal addresses?
Not by default: requests to loopback, private, link-local and cloud-metadata addresses are refused. An environment variable or a `CrawlConfig` setting lifts that.
Discussion
No questions or reviews yet.
Related
PluginsAn open-source crawler that turns websites into clean Markdown for AI agents: install from PyPI, run in Docker and connect to Claude Code over MCP
A Rust server that turns pages into Markdown or JSON: search, scrape, site map, crawl and PDF through MCP, a CLI and a Firecrawl-compatible API
Official Firecrawl MCP server: web search, page scraping to Markdown, site crawling and in-page actions
MCP server that reads sites on your machine and hands the agent short Markdown with links, keeping raw HTML out of context.
The Firecrawl command-line client with agent skills: web search, scraping, crawling, URL mapping and page interaction
An MCP server that opens pages in headless Chromium through Playwright and hands the agent the main text as Markdown, even from JavaScript-heavy sites
README in short
The main README is about the Crawlberg library: one Rust core, 14 language bindings, BM25 filtering, robots.txt and sitemap handling, batch and streaming work. The assistants section lists plugin install for Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Pi and others. The plugin README describes seven skills and nine MCP tools.