webclaw

Most web scraping tools give your agent one of two bad outputs:

• a blocked page, login wall, or empty app shell
• raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate

webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.

webclaw turns a URL into clean content your tools can actually use.

bashCopy code
webclaw https://example.com --format markdown

mdCopy code
# Example Domain

This domain is for use in illustrative examples in documents.

You may use this domain in literature without prior coordination or asking for permission.

Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.

Install

Agent setup

The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:

bashCopy code
npx create-webclaw

The installer detects supported clients and configures the MCP server for you.

Homebrew

bashCopy code
brew tap 0xMassi/webclaw
brew install webclaw

Prebuilt binaries

Download macOS, Linux, and Windows binaries from GitHub Releases.

Windows (x64)

The prebuilt ZIP is the easiest installation and does not require Rust. On the v0.6.22 release page, download webclaw-v0.6.22-x86_64-pc-windows-msvc.zip. Extract it in File Explorer, open PowerShell in the extracted folder containing webclaw.exe, and run:

powershellCopy code
.\webclaw.exe --version
.\webclaw.exe https://example.com --format markdown

PowerShell requires the .\ prefix for programs in the current directory. Add that directory to your user PATH only if you want to run webclaw from other folders. If Windows reports a missing Visual C++ runtime DLL, install the Microsoft Visual C++ Redistributable for x64. The ZIP contains x64 binaries; Windows ARM64 is not covered by the native installation check.

To build from source on Windows, install Rust with the x86_64-pc-windows-msvc toolchain, Visual Studio Build Tools with Desktop development with C++ and the Windows SDK, Git, CMake 3.22 or newer, LLVM (including libclang.dll), and NASM. Open a fresh Developer PowerShell for Visual Studio after installation. Ensure CMake and NASM are on PATH and set LIBCLANG_PATH to your LLVM bin directory if bindgen cannot find it, for example:

powershellCopy code
$env:LIBCLANG_PATH = "C:\Program Files\LLVM\bin"
cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-cli
webclaw --version

The Cargo package is webclaw-cli; the executable is webclaw. These packages are installed from Git, not crates.io. A first source build can take several minutes. The Windows CI workflow checks the Git installation and published ZIP on a hosted runner with preinstalled build tools; it does not model a clean PC. Linux containers, including Docker on macOS, do not validate Windows support.

Docker

bashCopy code
docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

bashCopy code
cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-mcp

If building from source fails because native build tools are missing, install the platform prerequisites:

Quick Start

Scrape one page

bashCopy code
webclaw https://stripe.com --format markdown

Return LLM-optimized text

bashCopy code
webclaw https://docs.anthropic.com --format llm

Keep only the main content

bashCopy code
webclaw https://example.com/blog/post --only-main-content

Include or exclude selectors

bashCopy code
webclaw https://example.com \
  --include "article, main, .content" \
  --exclude "nav, footer, .sidebar, .ad"

Crawl a documentation site

bashCopy code
webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50

Workflow examples

• HTML to Markdown for RAG
• Firecrawl-compatible API
• MCP web scraping
• Proxy-backed crawling with ColdProxy
• Cloudflare diagnostics

Extract brand assets

bashCopy code
webclaw https://github.com --brand

Compare a page over time

bashCopy code
webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.json

MCP Server

webclaw ships with an MCP server for AI agents.

Zero-install — point any MCP client at the npx launcher:

jsonCopy code
{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

Or run npx create-webclaw to auto-detect your AI tools and write their configs for you.

Then ask your agent things like:

textCopy code
Scrape these competitor pricing pages and summarize the differences.

textCopy code
Crawl this documentation site and prepare clean context for a RAG index.

textCopy code
Extract the brand colors, fonts, and logos from this company website.

Use as an agent skill

Add webclaw to Claude Code, Cursor, Windsurf, and other MCP agents in one command:

bashCopy code
npx skills add 0xMassi/webclaw-skill

Your agent gets scrape, crawl, map, extract, summarize, diff, brand, and search as native tools. Most sites extract locally with no API key. Set WEBCLAW_API_KEY to handle bot-protected and JavaScript-rendered pages.

Find it on skills.sh.

Tools

SDKs

bashCopy code
npm install @webclaw/sdk
pip install webclaw
go get github.com/0xMassi/webclaw-go

tsCopy code
import { Webclaw } from "@webclaw/sdk";

const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! });

const page = await client.scrape({
  url: "https://example.com",
  formats: ["markdown"],
  only_main_content: true,
});

console.log(page.markdown);

pythonCopy code
from webclaw import Webclaw

client = Webclaw(api_key="wc_your_key")

page = client.scrape(
    "https://example.com",
    formats=["markdown"],
    only_main_content=True,
)

print(page.markdown)

bashCopy code
curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer $WEBCLAW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "only_main_content": true
  }'

Output Formats

Local First, Hosted When Needed

The CLI and MCP server work locally without an account for the core extraction path.

Use the hosted API at webclaw.io when you need:

• protected-site access without managing infrastructure
• JavaScript rendering
• async crawl and research jobs
• web search
• watches and production usage tracking
• SDKs for application code

bashCopy code
export WEBCLAW_API_KEY=wc_your_key

webclaw https://example.com --cloud

What You Can Build

Architecture

textCopy code
webclaw/
  crates/
    webclaw-core     HTML to markdown, text, JSON, and LLM-ready output
    webclaw-fetch    Fetching, crawling, batching, and mapping
    webclaw-llm      Local and hosted LLM provider support
    webclaw-pdf      PDF text extraction
    webclaw-mcp      MCP server for AI agents
    webclaw-cli      Command-line interface

webclaw-core is pure extraction logic: no network I/O, small surface area, and usable independently from the fetching layer.

Configuration

Contributing

The most useful contributions right now are practical and small:

• add examples for real agent and RAG workflows
• improve SDK snippets
• report pages that extract poorly
• add failing fixtures for messy HTML
• improve docs for MCP clients and local setup
• test the CLI on more Linux/macOS environments

Good first places to start:

• Good first issues
• Open a bug report
• Start a discussion

If a page extracts badly, include:

textCopy code
URL:
Command or API request:
Expected output:
Actual output:
Format used: markdown / llm / text / json / html
CLI, MCP, SDK, or API:

Please remove secrets, cookies, private tokens, and customer data from logs before posting.

Strategic Partner

Infrastructure Partner

Studio Partners

Community Plugins

Third-party plugins that integrate webclaw with AI agent platforms:

Built a webclaw integration? Open a PR to add it here.

Contributors

Thanks to everyone improving webclaw through issues, examples, docs, bug reports, and pull requests.

Star History

License

AGPL-3.0