Searcharvester workflow
Searcharvester runs your own search for agents: SearXNG, page extraction to markdown and a team for deep research.
- 283 ★ on the repository
- Last commit
- AGPL-3.0
Extended access
Why this level
- The service sends search queries to public engines and the model from `.env`.
- The stack listens on local ports and needs protection if exposed.
Install
- Works in
Manual install
git clone https://github.com/vakovalskii/searcharvester.gitThen `docker compose up -d` after configuring `config.yaml` and `.env`. The UI opens at `http://localhost:9762`.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The service sends search queries to public engines and the model from `.env`.
- The stack listens on local ports and needs protection if exposed.
This is third-party code. Review the repository files before installing.
What it does
Searcharvester is a self-hosted search stack for AI agents started with `docker compose up -d`. The adapter provides a Tavily-compatible `/search` built on SearXNG with a hundred engines, `/extract` turns a page into clean markdown through trafilatura, readability and Defuddle, and `/research` runs an agent team that answers with a cited report. The web UI on port 9762 shows jobs, agent roles and their models, progress, sources and media. Agents use three skills from the `hermes_skills` folder: search, extract and deep research. Every POST, PUT and DELETE request carries an `X-Searcharvester-Client: 1` header.
Who it is for. Developers and researchers who want to give an agent web search without paid service keys and keep queries on their own server.
Good fit when
- An agent needs Tavily-compatible search with no quotas or per-request fees.
- A page must become clean markdown with pagination.
- A question calls for a cited report and you want to watch progress in a UI.
- SearXNG engines and proxies are set up in one place.
Not a fit when
- Docker and room for several containers are missing.
- You need a single search MCP server: the stack runs as an HTTP service.
- Queries cannot be sent to the public search engines SearXNG relies on.
Example request
Find out what trafilatura is and give one paragraph with a source through the local /research.Limitations
`/research` needs any OpenAI-compatible endpoint and a key in `.env`, and model costs remain yours. Search depends on public engines, so SearXNG may get blocks and captchas. Change `server.secret_key` to a string of 32 or more characters and set a token for search settings before starting. Ports listen on 127.0.0.1, so add authentication if exposing them. The last commit is from September 2026. Documentation is in Russian, English and Chinese.
How to disable. Stop the stack with `docker compose down` and delete the project folder with `.env` and `config.yaml`.
FAQ
Are keys needed for search?
No. Search and page extraction run with no keys or quotas. A key is only needed for `/research`, where a model works.
What does `/research` return?
A job with a `job_id`. Poll `GET /research/{job_id}` until completed to get a markdown report with citations.
Why the `X-Searcharvester-Client` header?
Every POST, PUT and DELETE to the adapter must carry it or be rejected. It protects against foreign requests.
Where do I watch the agents?
In the web UI on port 9762: jobs, a model per role, progress, the report and sources.
Which skills do the agents use?
Three: `searcharvester-search`, `searcharvester-extract` and `searcharvester-deep-research` from the `hermes_skills` folder.
Discussion
No questions or reviews yet.
Related
WorkflowsMCP server for web search and crawling: SearXNG and You.com plus clean Markdown from pages, sites and logged-in areas
A web research skill: SearXNG searches, Camofox reads pages and CloakBrowser opens sites that check their visitors
SearXNG AI Kit gives agents web search across 180+ engines through a CLI and an MCP server, with no search API key
A skill for web search, X search and page fetching for models without web access: no signup or key, the answer is JSON with source links
Three search skills: four-source search scored by intent, deep thread tracing and clean Markdown extraction from pages.
AutoSearch gives an agent search over 40 channels, from arXiv to Chinese social media, and returns sources with links.
README in short
The README covers Docker Compose startup, three endpoints, the web UI, the research agent team and security. Documentation exists in three languages.