searxNcrawl MCP
MCP server for web search and crawling: SearXNG and You.com plus clean Markdown from pages, sites and logged-in areas
- 189 ★ on the repository
- Last commit
Extended access
Why this level
- The server goes online and downloads pages and a captured sign-in session crawls under your account
- HTTP mode without an authenticating proxy is open to anyone who can reach the port
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add mcp/searxncrawlDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The server goes online and downloads pages and a captured sign-in session crawls under your account
- HTTP mode without an authenticating proxy is open to anyone who can reach the port
This is third-party code. Review the repository files before installing.
What it does
searxNcrawl gives an agent two abilities: search the web and fetch page content in clean form. Search runs in parallel through your SearXNG instance and optionally the You.com API, results are merged with their source and filtered by language and period. Crawling is built on Crawl4AI: one or several pages become Markdown or JSON without repeated blocks and with links on demand, while whole sites are fetched with depth and page limits, handy for documentation. For logged-in areas the session is captured once with `crawl-capture` and then crawling runs under that account. The server works over stdio or HTTP with Host and Origin checks and has `crawl`, `search` and `crawl-capture` commands and a Python API. The `uvx searxncrawl setup` command finds Claude Code, Codex, Gemini CLI, Cursor, VS Code and Claude Desktop and configures them after confirmation.
Who it is for. People who feed an agent fresh documentation and search results and want to keep search on their own SearXNG without paid keys.
Good fit when
- An agent looks for a library quick start page, opens the official one and retells how to run
- A site documentation is fetched whole as Markdown for questions over a knowledge base
- Search must go through your own SearXNG and not through someone else paid API
- A logged-in section needs a pass: the sign-in session is captured once and then crawled
- One `uvx searxncrawl setup` command configures several agents at once
Not a fit when
- You need to crawl sites that defend against automated requests: the server is not built for that
- A single search API request is enough: calling it directly is simpler
- There is no room for a browser: the first crawl downloads Chromium of about 550 MB
Example request
Find the Crawl4AI quick start page, open the official one and briefly retell how to run a first crawlMCP
- Transport
- stdio, http
- Authentication
- not required
Environment variables
SEARXNG_URL- URL of your SearXNG instance.
YDC_API_KEYsecret- You.com API key for search.
Limitations
Search needs at least one provider: a SearXNG instance with JSON output enabled or a `YDC_API_KEY` for You.com, while crawling works without them. The first crawl downloads Playwright Chromium, about 550 MB. The uv manager is required. For shared HTTP access put an authenticating TLS proxy in front and set `FASTMCP_HTTP_HOST_ORIGIN_PROTECTION=true`, since allowlists alone do not turn protection on. Sessions for logged-in areas stay with you and such sites should be crawled only with the owner permission.
How to disable. Run `uvx searxncrawl setup --remove`: it undoes the client configuration changes it made, backing files up first.
FAQ
How do I connect searxNcrawl to Claude Code?
Run `claude mcp add searxncrawl --env SEARXNG_URL=http://localhost:8888 -- uvx searxncrawl`. Or run `uvx searxncrawl setup` and the wizard configures every client it finds.
Do I need my own SearXNG?
The `search` tool needs at least one provider: SearXNG or a You.com key. Page crawling works without them. SearXNG can be run in Docker per `docs/usage/installation.md`.
How do I crawl pages behind a login?
Capture the session once with `crawl-capture` and crawling then runs under that account. Do this only on sites where you have the right to access.
What happens on the first crawl?
Playwright Chromium of about 550 MB downloads once. After that crawling runs with no download.
Which run modes exist?
Native MCP over stdio (default), over HTTP through `uvx searxncrawl --transport http` or Docker Compose and an optional adapter for the Arcade gateway.
Discussion
No questions or reviews yet.
Related
MCP serversSearXNG AI Kit gives agents web search across 180+ engines through a CLI and an MCP server, with no search API key
An MCP server for private web search through your SearXNG instance: search with pagination and filters, page and PDF reading
An open-source crawler that turns websites into clean Markdown for AI agents: install from PyPI, run in Docker and connect to Claude Code over MCP
A web search MCP server that gives the agent full page text at once: Stack Overflow answers, GitHub threads, arXiv papers
A deep research MCP server: multi-source search, result fusion and cited synthesis through a Qwen3 model on OpenRouter
Searcharvester runs your own search for agents: SearXNG, page extraction to markdown and a team for deep research.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| searxNcrawl (this card) | no | not required | 189 | |
| SearXNG MCP Server | no | not required | 1,359 | search through SearXNG only; this one adds page and site crawling to Markdown and logged-in areas |
| Crawl4AI | yes | API key | 87,271 | a crawling library; this is a ready MCP server with search and a client setup wizard |
README in short
The README describes three abilities (search, page and site crawling, logged-in areas), shows installation for the main clients and the `setup` wizard, then a diagram of how it works with SearXNG and Crawl4AI. A connection mode table closes it.