DigCrawl skill
A local content crawler for agents: posts, video and pages from public sources and your sessions. Skill, MCP, CLI.
- 244 ★ on the repository
- Last commit
- MIT
Broad access
Why this level
- The tool works with your authorized sessions on platforms and reads pages closed to guests.
- Collection is subject to platform rules, and a violation can get the account blocked.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/digcrawlDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The tool works with your authorized sessions on platforms and reads pages closed to guests.
- Collection is subject to platform rules, and a violation can get the account blocked.
This is third-party code. Review the repository files before installing.
What it does
DigCrawl reads links an agent cannot open on its own: articles and posts from WeChat, Xiaohongshu, Weibo, Zhihu, X, Reddit, Threads and Instagram, videos from Douyin, Toutiao, Bilibili, Kuaishou and YouTube, plus plain pages, RSS, Atom and sitemaps. For sources that need sign-in it uses your authorized local session: on `LOGIN_REQUIRED` the agent opens a separate Chrome for Testing window where you sign in. For video it takes available captions, and when configured it transcribes speech and recognizes on-screen text. Results are saved as Markdown, HTML, JSON and media files: `digcrawl collect "<link>" --format md,html,json`, in batches with `digcrawl collect --file urls.txt --out ./collect-output`. `digcrawl watchlist run --once` or `serve` monitors sources and sends notifications. It connects three ways: a skill for Codex and Claude Code, an MCP server and a Go CLI. Everything runs locally.
Who it is for. Researchers and editors who give an agent links to social networks and video platforms, including Chinese ones, for summaries, comparison and monitoring.
Good fit when
- The agent needs a link to a Xiaohongshu post or WeChat article that it cannot open.
- Analyzing a YouTube or Bilibili video requires the description, chapters and captions.
- A list of links from a file is saved as a batch in Markdown and JSON.
- A set of sources is monitored continuously with notifications about updates.
Not a fit when
- You need to collect closed content without your own sign-in: the tool works with public sources and sessions you authorized.
- You need a cloud service with no install on your machine.
- The source is not on the supported list and has no RSS: there is no reader for it.
Example request
Read and summarize this Xiaohongshu post link and save the text and images as Markdown.Limitations
Sources that need sign-in require your own authorized session in a separate Chrome for Testing window, and its data stays on your machine. Working with platforms is subject to their rules and your national laws, and the user is responsible for collection. Speech transcription and on-screen text recognition are enabled by configuration and need their own processors. The `npx` install needs Node.js 20.19 or newer, and the macOS and Linux script puts the program in `~/.local/bin`. The license is MIT, the last commit dates from November 2026.
How to disable. Delete the program from `~/.local/bin` and the skill from the agent skills folder, and remove the MCP entry from the configuration. Sign-in profiles and collected results sit separately and are deleted by hand.
FAQ
How do I install DigCrawl?
Run `npx digcrawl-skill@latest install` on Node.js 20.19 or newer. On macOS and Linux there is also the `install.sh` script from the README, which needs no Go.
Which agents connect?
The install configures Codex automatically and installs the Claude Code skill when it is detected. For others there is an MCP guide in `.agents/skills/digcrawl/mcp/README.md`.
What should I do on LOGIN_REQUIRED?
Ask the agent to open the sign-in window for the platform, for example YouTube. Sign in yourself, after which collection repeats with your session.
What formats are results saved in?
Markdown, HTML, JSON and media files.
How do I analyze a local video?
With `digcrawl video ./meeting.mp4`, which uses the configured transcription and recognition processors.
Discussion
No questions or reviews yet.
Related
SkillsThe news-extractor skill pulls the title, text, images and metadata from a link on 12 news platforms and returns JSON or Markdown, with no browser
CLI and skill that let an agent read and search Twitter, Reddit, YouTube, GitHub, Bilibili and Xiaohongshu without paid APIs
An open-source crawler that turns websites into clean Markdown for AI agents: install from PyPI, run in Docker and connect to Claude Code over MCP
A skill that reads a link and hands the agent the full text as Markdown, picking a separate route for WeChat pages, Feishu docs, PDFs and arXiv papers
A skill that turns a link into clean Markdown via Scrapling with a Jina Reader fallback, incl. WeChat, Juejin and CSDN articles
MCP server for web search and crawling: SearXNG and You.com plus clean Markdown from pages, sites and logged-in areas
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| DigCrawl (this card) | no | 244 | ||
| Crawl4AI | yes | API key | 87,271 | Crawl4AI crawls web pages, while DigCrawl reads social networks and video platforms, including Chinese ones, through your session. |
| SocialCrawl MCP | yes | key is optional | 28 | SocialCrawl MCP serves social data through a service, while DigCrawl runs locally with your sessions and adds a CLI and monitoring. |
README in short
A skill, MCP server and Go CLI for reading posts and videos from 15 platforms, including Chinese ones, with Markdown export and monitoring.