NewsCrawler skill
The news-extractor skill pulls the title, text, images and metadata from a link on 12 news platforms and returns JSON or Markdown, with no browser
- 631 ★ on the repository
- Last commit
- GPL-3.0
Extended access
Why this level
- The script itself requests the addresses you give it and saves pages to disk
- For protected tweets the cookie is passed on the command line and stays in the shell history
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/newscrawlerDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The script itself requests the addresses you give it and saves pages to disk
- For protected tweets the cookie is passed on the command line and stays in the shell history
This is third-party code. Review the repository files before installing.
What it does
NewsCrawler packages the `news-extractor` skill: the agent gets an article link and a local script pulls out the title, paragraphs, images, videos and the author and publication time. The script detects the platform from the address. It supports 12 sites: WeChat, Toutiao, NetEase, Sohu, Tencent News, BBC, CNN, X (Twitter), Lenny's Newsletter, Naver Blog, Detik and Quora. The output has one shape: JSON with the fields `title`, `news_url`, `news_id`, `meta_info`, `contents`, `texts`, `images`, `videos`, or Markdown with an article-info block, the body and a media list. The agent opens no browser and does not read the whole page into context, which leaves more room for summarizing and comparing. The author's measurement on one Detik article gave 1,730 tokens for the JSON against 59,788 for the rendered page. The command is `uv run scripts/extract_news.py "URL"`, with `--output`, `--format json|markdown` and `--list-platforms` options. Protected tweets take a `--cookie` parameter. The repository also holds an MCP server, FastAPI, a web UI and Docker, none of which the skill needs.
Who it is for. For analysts, editors and researchers who have an agent read, summarize and compare news from Chinese and international sources.
Good fit when
- You need to summarize an article from WeChat, Toutiao or NetEase without copying the text out of a browser by hand
- The agent is preparing a digest from a dozen BBC, CNN and Naver links and needs one JSON per article with paragraphs and images
- You do not want a whole page loaded into context: the agent needs only the title, paragraphs and a list of images
- You want the supported platform list from `--list-platforms` to check beforehand whether a link will work
Not a fit when
- The link points outside the 12 supported platforms: the script will report the platform as unrecognized
- You need material behind a login: you must supply the cookie yourself, and the skill describes that route only for protected tweets
- You need a permanent service with a queue and an API: the repository has an MCP server and FastAPI for that, but you deploy them separately
- You need another output format: the script returns only JSON and Markdown
Example request
Extract the text and images from these BBC and NetEase articles, save them as Markdown in ./news and compare the two stories brieflyLimitations
Only the 12 listed platforms are supported. Site structure, login rules and bot defenses change, so when something fails the author advises checking the link shape, network access and cookies first, then the platform adapter. The script makes the network calls itself and uses `curl_cffi`, which imitates a browser in requests. Availability of the platforms from Russia varies, so check the sources you need in advance.
You need Python, the `uv` manager and a one-time dependency install with `uv sync` in the skill directory; all scripts run through `uv run` rather than `python` directly. Dependencies are pydantic, requests, curl_cffi, tenacity, parsel and demjson3. Protected tweets take a cookie such as `auth_token` and `ct0` on the command line, and these are secrets. The code is released under GPL-3.0. The author gives a token measurement for a single article only and notes that other sites will give different results.
How to disable. Delete the `news-extractor` folder from the agent's skills directory, for example `~/.claude/skills/`. The virtual environment created by `uv sync` lives inside that folder and goes with it. Saved JSON and Markdown files stay in the `--output` directory, `./output` by default.
FAQ
How do I install news-extractor in Claude Code?
Run `npx skills add NanmiCoder/NewsCrawler --skill news-extractor -g -a claude-code -y`. Then run `uv sync` once in the skill directory and check `uv run scripts/extract_news.py --list-platforms`.
Which platforms are supported?
Twelve: WeChat, Toutiao, NetEase, Sohu, Tencent News, BBC, CNN, X (Twitter), Lenny's Newsletter, Naver Blog, Detik and Quora. The platform is detected from the link, and SKILL.md shows the address shapes.
What does the script return?
By default two files: `{news_id}.json` with the title, metadata, paragraphs, images and videos, and `{news_id}.md` with the same as Markdown. `--format json` or `--format markdown` keeps just one.
How do I fetch a protected tweet?
Pass a cookie with `--cookie "auth_token=...; ct0=..."`. Public tweets work without logging in. Never publish or commit the cookie: the project asks you not to include it even in tickets.
Do I need the MCP server or Docker?
No. The author notes that the skill needs neither MCP, FastAPI, the web UI nor Docker; they serve shared-service scenarios.
Discussion
No questions or reviews yet.
Related
SkillsA skill that reads a link and hands the agent the full text as Markdown, picking a separate route for WeChat pages, Feishu docs, PDFs and arXiv papers
CLI and skill that let an agent read and search Twitter, Reddit, YouTube, GitHub, Bilibili and Xiaohongshu without paid APIs
A skill that gathers 30 days of discussion on eight Chinese platforms, ranks it and writes a cited report, and can also show the cross-platform hot lists
A skill that routes a question to the right AI platform and, via browser automation, collects the answer from its native search
A Twitter/X CLI with an agent skill: feeds, bookmarks, search and profiles in the terminal, plus posting, replies and likes with no API keys
A skill for web search, X search and page fetching for models without web access: no signup or key, the answer is JSON with source links
Comparison with similar tools
Entries that share the main search query, and how they differ.
| Entry | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| NewsCrawler (this entry) | no | 631 | ||
| Firecrawl CLI | no | 661 | general scraping, search and crawling of any site through the Firecrawl service, while this skill parses articles from 12 platforms with local code | |
| Firecrawl MCP Server | yes | API key | 7,668 | Firecrawl's MCP server for scraping any page, whereas here adapters are written in advance for news platforms |
README in short
The Chinese description presents `news-extractor` as the repository's main product. It shows the 12 platforms, a comparison with the ordinary browser route, a token measurement on a Detik article from 20 July 2026 and install routes through `npx skills`. Below it explains the repository layout, contribution rules and the extra modes: MCP, FastAPI, a web UI and Docker.
README badge
Original SKILL.md
--- name: news-extractor description: 新闻站点内容提取。支持 12 个平台：微信公众号、今日头条、网易新闻、搜狐新闻、腾讯新闻、BBC News、CNN News、Twitter/X、Lenny's Newsletter、Naver Blog、Detik News、Quora。当用户需要提取新闻内容、抓取公众号文章、爬取新闻、或获取新闻JSON/Markdown时激活。 license: GPL-3.0 --- # News Extractor Skill 从主流新闻平台提取文章内容，输出 JSON 和 Markdown 格式。 **标准兼容、独立可迁移**：本目录遵循 Agent Skills 规范，包含完整的 `SKILL.md`、脚本与依赖，不依赖 NewsCrawler 的其他模块。通过兼容的 Skills 安装器安装后，运行一次 `uv sync` 即可使用。 运行环境需要 Python 3.9+、`uv` 和目标站点的网络访问。 ## 支持平台 (12) ### 中文平台 | 平台 | ID | URL 示例 | |------|-----|----------| | 微信公众号 | wechat | `https://mp.weixin.qq.com/s/xxxxx` | | 今日头条 | toutiao | `https://www.toutiao.com/article/123456/` | | 网易新闻 | netease | `https://www.163.com/news/article/ABC123.html` | | 搜狐新闻 | sohu | `https://www.sohu.com/a/123456_789` | | 腾讯新闻 | tencent | `https://news.qq.com/rain/a/20251016A07W8J00` | ### 国际平台 | 平台 | ID | URL 示例 | |------|-----|----------| | BBC News | bbc | `https://www.bbc.com/news/articles/c797qlx93j0o` | | CNN News | cnn | `https://edition.cnn.com/2025/10/27/uk/article-slug` | | Twitter/X | twitter | `https://x.com/user/status/123456789` | | Lenny's Newsletter | lenny | `https://www.lennysnewsletter.com/p/article-slug` | | Naver Blog | naver | `https://blog.naver.com/username/123456` | | Detik News | detik | `https://news.detik.com/internasional/d-123456/slug` | | Quora | quora | `https://www.quora.com/question/answers/123456` | ## 依赖安装 本 skill 使用 uv 管理依赖。首次使用前需要安装： ```bash cd <Agent Skills 目录>/news-extractor uv sync ``` **重要**: 所有脚本必须使用 `uv run` 执行，不要直接用 `python` 运行。 ### 依赖列表