Open SEO Crawler: an SEO crawler with MCP for agents
Open SEO Crawler
A free local SEO crawler with a site audit and AI bot check: agents start crawls and read the issues
- 82 ★ on the repository
- Last commit
Extended access
Why this level
- The crawler sends many requests to a site and can load it, and some requests carry search and AI robot user agents
- The crawler's daily auto-update runs new code on your computer
Install
- Works in
Manual install
claude mcp add --scope user open-seo-crawler -- ~/open-seo-crawler/venv/bin/python3 ~/open-seo-crawler/mcp_server.pyInstall the crawler first with the one-liner from the README, and it must run on port 5002.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The crawler sends many requests to a site and can load it, and some requests carry search and AI robot user agents
- The crawler's daily auto-update runs new code on your computer
This is third-party code. Review the repository files before installing.
What it does
Open SEO Crawler is an open-source SEO crawler that runs on your computer. It crawls a site fast, accounts for the CMS and looks for technical issues: indexability and noindex in meta tags and X-Robots-Tag, JSON-LD errors named by block and line, redirect chains, duplicate titles, deep pages, crawl budget traps and a set of reports exportable to xlsx. The author compares it with Screaming Frog and Sitebulb. Beyond the audit there is an AI visibility check: after a crawl a sample of pages is requested as a browser, as Googlebot, Bingbot, Applebot and as AI crawlers with published user agents such as GPTBot and PerplexityBot. Blocks, CDN challenges, redirects and differences in title or text are flagged. The robots.txt check shows blocked AI crawlers. The repository includes an MCP server `mcp_server.py` and a console utility `osc.py`. An agent starts a crawl, follows it, reads the summary, filters pages by issue, opens a page with its inlinks, compares two crawls and exports xlsx. The crawler itself calls no AI.
Who it is for. SEO specialists, web developers and site owners who want a technical audit on their own computer and to go through the results with an agent.
Good fit when
- You want to crawl your own site and get the ten most important fixes
- You check whether Googlebot and AI crawlers see your pages' content
- You want to compare two crawls before and after a release
- You need to find crawl budget traps and get rules for robots.txt
Not a fit when
- You need a cloud service with history and teamwork: the crawler is local
- You want to crawl other people's sites without permission: the tool is meant for auditing your own sites
- You need backlink and ranking analysis: the crawler looks at technical page issues
Example request
Crawl example.com for 100 pages and give the ten fixes that matter most, then check whether GPTBot sees the pages.MCP
- Transport
- stdio
- Authentication
- not required
Limitations
An installed crawler is required: one line on Linux, macOS or Windows, and on macOS 11 Big Sur or newer through Homebrew and `launchd`. The installers update the crawler daily and start it at boot, and the MCP server and `osc.py` work only while the crawler runs on port 5002.
The AI bot check requests pages with search and AI robot user agents, so do it only on your own sites or with the owner's permission. For ClaudeBot and others without a published user agent only robots.txt is checked. JavaScript sites need Playwright. The comparison table with paid crawlers was compiled by the author.
How to disable. Remove the server with `claude mcp remove open-seo-crawler`. The crawler itself stops and uninstalls with the uninstall script from the `~/open-seo-crawler` directory, taking its autostart with it.
FAQ
How do I connect Open SEO Crawler to Claude Code?
Install the crawler, then run `claude mcp add --scope user open-seo-crawler -- ~/open-seo-crawler/venv/bin/python3 ~/open-seo-crawler/mcp_server.py` and check `/mcp`.
Does the MCP server open a network port?
No. The app starts it as a local program and talks over stdin and stdout. Port 5002 belongs to the crawler itself.
What does the AI crawler check do?
After a crawl a page sample is requested as a browser, Googlebot, Bingbot, Applebot and AI crawlers with public user agents. Blocks, content differences and noindex for one bot are flagged.
What is the license?
MIT, free with no page limit.
Does the crawler call AI itself?
No. The crawler calls no AI and sends nothing out. Your assistant makes the calls.
Discussion
No questions or reviews yet.
Related
MCP serversAn open MCP server and agent skill set for SEO: keyword research, competitors, backlinks and site audits powered by DataForSEO
A plugin with an MCP server and skills: Claude Code crawls a site, finds broken links and SEO and GEO errors and suggests fixes.
Screaming Frog MCP: SEO crawl data for agents
Screaming Frog MCP
An unofficial headless MCP server for Screaming Frog: run crawls, export and analyze data with nine tools and no file write access
crawlseo MCP compares the latest 28 days of site metrics with the preceding 28 days, alongside results from the latest crawl.
An MCP server with SEO data: backlinks, keyword difficulty and traffic estimates from the free Ahrefs tools
All-In-One Free SEO Tool
All-In-One-Free-SEO-Tool
A self-hosted SEO suite with an MCP server and five skills for audits, keywords and site fixes in Claude Code and Cursor.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| Open SEO Crawler: an SEO crawler with MCP for agents (this card) | no | not required | 82 | |
| Open SEO MCP Skills | no | 9,422 | a set of skills and MCP for SEO with no crawler of its own | |
| seo-audit-skill | no | 785 | an audit checklist skill with no site crawl | |
| Lighthouse MCP | no | not required | 219 | measures the speed of individual pages and does not crawl a site |
README in short
The README compares the crawler with paid alternatives, lists the checks, one-line installers for three systems and launch details. A separate section shows use from an AI assistant: the MCP server, the `osc.py` utility, tools such as `check_robots` and `detect_cms` and a sample request.