SkillFoxx

Web Content Fetcher skill

A skill that turns a link into clean Markdown via Scrapling with a Jina Reader fallback, incl. WeChat, Juejin and CSDN articles

  • 613 ★ on the repository
  • Last commit
Skill

Extended access

Why this level

  • The `--stealth` mode starts a headless browser for sites with anti-scraping protection, which may break those sites' rules.
  • The fallback path sends the page URL to the third-party Jina Reader service.
All reasons and checks

shirenchuang/web-content-fetcherAuthor:shirenchuang

Install

In your terminal, with SkillFoxx CLI

npx skillfoxx add skills/web-content-fetcher

Detects the agents on your machine, shows the access level and pins the version.

Other ways to install

Access and checks

How we review

Extended accessThe tool runs code or makes network calls and only reads external data.

  • The `--stealth` mode starts a headless browser for sites with anti-scraping protection, which may break those sites' rules.
  • The fallback path sends the page URL to the third-party Jina Reader service.

This is third-party code. Review the repository files before installing.

What it does

The `web-content-fetcher` skill takes a link, extracts the main text of the page and returns it as clean Markdown with headings, links, images, lists and code blocks. The agent runs the `scripts/fetch.py` script: it first makes a fast HTTP request that takes 1 to 3 seconds, and if too little text came back it switches on its own to `--stealth` mode with a headless browser, which spends 5 to 15 seconds on JavaScript pages. The domain table in the skill picks the mode up front: WeChat, Zhihu and Juejin articles go through `--stealth`, while sspai, CSDN and most blogs are fine with the fast mode. If Scrapling is not installed or fails, the agent falls back to Jina Reader at `https://r.jina.ai/<url>`. The script takes a character limit as the second argument and a `--json` flag that adds the mode, selector and text length to the output.

Who it is for. Researchers and editors who ask Claude Code to read an article from a link, especially on Chinese platforms, and return it as Markdown.

Good fit when

  • The agent has to read a WeChat article from a link and summarize it.
  • A page returns raw HTML while notes need headings, links and code.
  • Documentation or a blog is saved as Markdown for a knowledge base.
  • A site renders content with JavaScript and a fast request returns an empty shell.

Not a fit when

  • The page is behind a login, like Xiaohongshu: the script cannot open it without a session.
  • The site sits behind a Cloudflare check: on Product Hunt the script gets stuck on it.
  • Page URLs must not go to external services: the fallback path sends the address to Jina Reader.

Example request

Read the article at this WeChat link, return it as Markdown with images and list the main points.

Limitations

Python 3.10 or newer and the `scrapling` and `html2text` packages are required; on the system Python of macOS and Linux they are installed with `--break-system-packages` or in a virtual environment. The `--stealth` mode starts a headless browser and is noticeably slower. The Jina Reader fallback is limited to 200 requests per day on the free tier and cannot open WeChat. The repository has no license file although the README calls the project MIT. The last commit is from March 2026.

How to disable. Delete the `web-content-fetcher` folder from the skills directory, for example `~/.claude/skills/`, and optionally run `pip uninstall scrapling html2text`.

FAQ

How do I install web-content-fetcher in Claude Code?

Run `npx skills add https://github.com/shirenchuang/web-content-fetcher --skill web-content-fetcher` and install dependencies with `pip install scrapling html2text`. Copying the folder into `~/.claude/skills/` also works.

When should stealth mode be used?

For pages built with JavaScript: WeChat, Zhihu, Juejin. On other sites the script switches over from the fast mode on its own when there is too little text.

How do I limit output size?

Pass a number as the second argument, for example `python3 scripts/fetch.py https://example.com/article 15000`. Without it the limit is 30,000 characters.

How do I get metadata along with the text?

Add the `--json` flag: the output then contains `url`, `mode`, `selector` and `content_length`.

What if Scrapling is unavailable?

The agent requests `https://r.jina.ai/<url>` through `web_fetch`. It is fast but does not open WeChat and some Chinese sites.

Discussion

No questions or reviews yet.

Skills

A skill that reads a link and hands the agent the full text as Markdown, picking a separate route for WeChat pages, Feishu docs, PDFs and arXiv papers

SkillExtended access512

The news-extractor skill pulls the title, text, images and metadata from a link on 12 news platforms and returns JSON or Markdown, with no browser

SkillExtended access639

Skill instead of WebFetch reads a page and returns only the blocks that answer the question, verbatim with heading paths.

SkillExtended access55

A skill that collects fresh news from dozens of tech, finance and AI sources with no API keys and prepares a digest in Chinese.

SkillExtended access1.3K

A skill that searches WeChat public-account articles by keyword through Sogou and returns titles, links and dates as JSON

SkillBroad access276

Comparison with similar tools

Cards that share the main search query, and how they differ.

ToolOfficialAuthenticationRepository starsMain difference
Web Content Fetcher (this card)no613
jina-cli: web reading and search via Jina Readerno311a skill and utility around Jina Reader; here the main path is Scrapling with a switch to browser mode and Jina stays a fallback
Fetcher MCPnonot required1,088an MCP server that opens pages in Chromium through Playwright; here a skill with a script and per-domain mode choice

README in short

The Chinese-language description presents the project as a free text extraction tool focused on WeChat, Juejin and CSDN. It explains the extraction strategy, domain routes and a list of tested Chinese and foreign platforms, some marked with limitations.

README badge

Original SKILL.md

---
name: web-content-fetcher
description: >
  Extract article content from any URL as clean Markdown.
  Uses Scrapling script as primary method (with auto fast→stealth fallback),
  Jina Reader as alternative for simple pages.
  Preserves headings, links, images, lists, and code blocks.
  Use this skill whenever the user wants to fetch, read, extract, scrape, or summarize
  content from a URL — including blog posts, news articles, WeChat articles (微信公众号),
  documentation pages, or any web page. Also trigger when the user says things like
  "帮我读一下这篇文章", "抓取这个网页", "提取正文", or "read this page for me".
---

# Web Content Fetcher

Given a URL, return its main content as clean Markdown — headings, links, images, lists, code blocks all preserved.

## Extraction Strategy

Always try **one method per URL** — don't cascade blindly. Pick the right one upfront.

```
URL
 │
 ├─ 1. Scrapling script (preferred)
 │     Run fetch.py — check the domain routing table to decide fast vs --stealth.
 │     Works for most sites. Returns clean Markdown directly.
 │
 └─ 2. Jina Reader (fallback — only if Scrapling fails or dependencies not installed)
       web_fetch("https://r.jina.ai/<url>")
       Free tier: 200 req/day. Fast (~1-2s), good Markdown output.
       Does NOT work for: WeChat (403), some Chinese platforms.
```

### Scrapling script

```bash
python3 <SKILL_DIR>/scripts/fetch.py "<url>" [max_chars] [--stealth]
```

`<SKILL_DIR>` is the directory where this SKILL.md lives. Resolve it before calling the script.

The script has two modes built in:
- **Default (fast):** HTTP fetch, ~1-3s, works for most sites
- **`--stealth`:** Headless browser, ~5-15s, for JS-rendered or anti-scraping sites

Report a problem