XCrawl Skills
Five skills for the XCrawl API: scraping a single page, mapping a site's URLs, crawling a whole site and search, with cURL and Node examples
- 449 ★ on the repository
- Last commit
Broad access
Why this level
- The skills send requests to a third-party paid service and spend your account's credits
- Bulk crawling of other sites may violate their terms of use and data laws
- The API key sits in plain text in `~/.xcrawl/config.json`
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/xcrawl-skillsDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- The skills send requests to a third-party paid service and spend your account's credits
- Bulk crawling of other sites may violate their terms of use and data laws
- The API key sits in plain text in `~/.xcrawl/config.json`
This is third-party code. Review the repository files before installing.
What it does
XCrawl Skills is a set of five skills for working with XCrawl, a web-data service for search, scraping, URL mapping and site crawling. The `xcrawl` skill is the default entry and handles direct requests and single-page extraction. The others are narrower: `xcrawl-scrape` for structured data from a URL, `xcrawl-map` for URL discovery, `xcrawl-crawl` for bulk crawling with asynchronous results and `xcrawl-search` for query-based search with country and language controls. Each `SKILL.md` documents scenarios, request and response parameters and cURL and Node examples. The API base URL is `https://run.xcrawl.com`, and the main endpoints are `POST /v1/scrape`, `POST /v1/map`, `POST /v1/crawl` and `POST /v1/search`. Results of asynchronous jobs are fetched by ID with `GET`. The skills return the API response as it is, without retelling it. The `XCRAWL_API_KEY` key lives in the local file `~/.xcrawl/config.json` and goes in the `Authorization: Bearer` header. Sample requests from the README: fetch a page synchronously and return Markdown with links, collect only URLs under `/docs/` with a limit of 2000, start a crawl with depth 2 and limit 100 and wait for completion.
Who it is for. Developers of agents and analysts who need web data through a ready API: pages as Markdown, site maps and search results by country.
Good fit when
- A page has to be fetched synchronously and returned as Markdown with links
- From a site's documentation you want only the URLs under `/docs/`, capped by a number
- You need a bounded crawl of a site by depth and page limit with a wait for the result
- Search results must come for a specific country and language
- Data has to be extracted from a page by a `json_schema` or a prompt
Not a fit when
- You have no XCrawl key and do not plan to register: every request goes through a paid service
- You want your own crawler on your own servers: the skills only call the XCrawl API
- A site's terms forbid automated collection: you have to check permissions yourself
- The agent cannot run `curl` and `node`: the examples rely on those binaries
Example request
Use XCrawl to collect only URLs under /docs/ on this site with a limit of 2000 and return the listLimitations
The README has no install command, and the author does not describe installation: the generic `npx skills add xcrawl-api/xcrawl-skills` fits because the repository has a `SKILL.md`. The last commit dates from March 2026. An XCrawl key is required: `dash.xcrawl.com` offers a free plan with 1000 credits. `curl` and `node` must be available. The skills return the API response unprocessed, so the agent itself parses and cleans the data.
How to disable. Delete the `xcrawl*` skill folders from the agent's skills directory and the `~/.xcrawl/config.json` file holding the key. You can revoke the key itself in the dashboard at `dash.xcrawl.com`.
FAQ
Which skill should I pick for one page?
A direct request with no extras is handled by `xcrawl`, which calls scraping by default. If structured data and formats matter, use `xcrawl-scrape`.
How do I find out which pages a site has?
The `xcrawl-map` skill collects a site's URLs and helps plan the scope. Then `xcrawl-crawl` walks the chosen area and returns results through a job ID that you poll until the job finishes.
Where do I put the API key?
In the file `~/.xcrawl/config.json` as `{"XCRAWL_API_KEY": "<your_key>"}`. The skills read the key from this file and send it in the `Authorization: Bearer` header.
How do the skills hand the response to the agent?
Without retelling: by default the API response body is returned as it is. The normalized skill output carries a status, the request payload, the raw response or an async pair, job IDs and an error.
How many requests does the free plan give?
The README names a free plan of 1000 credits after registering at `dash.xcrawl.com`. How many credits each kind of request costs is in the XCrawl documentation.
Discussion
No questions or reviews yet.
Related
SkillsPortable skill for the Crawl4AI CLI and SDK: JavaScript-heavy pages, structured extraction by CSS schemas and batch crawling.
Five skills that teach an agent to wire Firecrawl search, scraping and page interaction into the code of your own application
Firecrawl skills catalog for Claude Code, Cursor and Codex: web search, scraping, crawling, page interaction and API integration
A skill for web search, X search and page fetching for models without web access: no signup or key, the answer is JSON with source links
A local content crawler for agents: posts, video and pages from public sources and your sessions. Skill, MCP, CLI.
A web research skill: SearXNG searches, Camofox reads pages and CloakBrowser opens sites that check their visitors
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| XCrawl Skills (this card) | yes | 449 | ||
| Firecrawl CLI | no | 684 | a command-line client for another service, Firecrawl, while the XCrawl skills work with its own API and key | |
| Crawl4AI | yes | API key | 87,271 | an open-source crawler you run yourself in Docker, while XCrawl is called as a cloud API |
README in short
The description presents XCrawl as a web-data service and lists the skills. Then come prerequisites (a key, `curl` and `node`), setting up the key file, choosing a skill, sample intents, a single input and output contract for different runtimes, shared conventions on addresses and links to the documentation.
README badge
Original SKILL.md
---
name: xcrawl
description: Use this skill as the default XCrawl entry point for direct XCrawl requests, including single-URL fetch, format selection, sync or async execution, and JSON extraction with prompt or json_schema.
allowed-tools: Bash(curl:*) Bash(node:*) Read Write Edit Grep
metadata: {"version":"1.0.2","openclaw":{"skillKey":"xcrawl","homepage":"https://www.xcrawl.com/","requires":{"localFiles":["~/.xcrawl/config.json"],"anyBins":["curl","node"]},"apiKeySource":"local_config"}}
---
# XCrawl
## Overview
This skill is the default XCrawl entry point when the user asks for XCrawl directly without naming a specific API or sub-skill.
It currently targets single-page extraction through XCrawl Scrape APIs.
Default behavior is raw passthrough: return upstream API response bodies as-is.
## Routing Guidance
- If the user wants to extract one or more specific URLs, use this skill and default to XCrawl Scrape.
- If the user wants site URL discovery, prefer XCrawl Map APIs.
- If the user wants multi-page or site-wide crawling, prefer XCrawl Crawl APIs.
- If the user wants keyword-based discovery, prefer XCrawl Search APIs.
## Required Local Config
Before using this skill, the user must create a local config file and write `XCRAWL_API_KEY` into it.
Path: `~/.xcrawl/config.json`
```json
{
"XCRAWL_API_KEY": "<your_api_key>"
}
```
Read API key from local config file only. Do not require global environment variables.
## Credits and Account Setup
Using XCrawl APIs consumes credits.
If the user does not have an account or available credits, guide them to register at `https://dash.xcrawl.com/`.