Web Scraping

Web Scraping

Overview

Two ways to scrape in the sandbox:

1. MCP tools (scrape_page, scrape_pages) — recommended for straight "give me this page's content". Synchronous, return dicts.
2. Direct Scrapling Python API — for CSS/XPath selectors, sessions, logins, and multi-page spiders. Async, returns Page objects with .css() / .xpath().

Quick fetches can run inline via ExecuteCode. For spiders, multi-URL crawls, or anything you'll iterate on, write the scraper to work/<task_name>/scraper.py and run it via Bash — edit-and-rerun beats resubmitting code.

MCP Tools

Import from tools.scrape. Synchronous — no await.

pythonCopy code
from tools.scrape import scrape_page, scrape_pages

Signatures

pythonCopy code
scrape_page(url: str, mode: str = "fast", extraction: str = "markdown",
            timeout: float = 30.0, solve_cloudflare: bool = False) -> dict

scrape_pages(urls: list[str], mode: str = "fast", extraction: str = "markdown",
             timeout: float = 30.0, solve_cloudflare: bool = False) -> dict

Parameters

Escalate modes only as needed: start fast, go to browser when the page needs JavaScript, stealth when you're getting blocked, and add solve_cloudflare=True only if stealth still returns a challenge page.

Return shape

scrape_page returns a flat dict:

pythonCopy code
{
    "url": "https://example.com",
    "status": 200,
    "title": "Example Domain",
    "content": "# Example Domain\n\nThis domain is for use in...",  # str
    "extraction": "markdown",
    "mode": "fast",
}

• content is a plain string, not a list — use it directly, never content[0] (that yields a single character).
• content is truncated to 400,000 chars.
• No .css() / .xpath() / .body / .headers / .cookies — for selectors use the direct Python API below, or parse extraction="html" with BeautifulSoup.

scrape_pages wraps them:

pythonCopy code
{
    "results": [ ... ],  # one entry per input URL, in input order
    "count": 3,
}

Errors

Errors are returned, never raised. Always check for "error" before reading content.

pythonCopy code
res = scrape_page(url="https://example.com")
if "error" in res:
    print(res["error"], res["detail"])
else:
    print(res["content"])

Per-URL errors — appear as {"error", "detail", "url"} entries inside scrape_pages["results"], or as the whole return of scrape_page:

Whole-call errors — the entire return is {"error", "detail"}, no results:

One bad URL never sinks a batch. scrape_pages always returns one entry per input URL, in input order — failures come back as error entries alongside the successes.

Examples

pythonCopy code
from tools.scrape import scrape_page, scrape_pages

# Single page → markdown
res = scrape_page(url="https://example.com")
if "error" not in res:
    print(res["title"], res["status"], len(res["content"]))

# JS-rendered page
res = scrape_page(url="https://spa-site.com", mode="browser", timeout=60)

# Bot-protected page
res = scrape_page(url="https://protected-site.com", mode="stealth", solve_cloudflare=True)

# Batch — split successes from failures
batch = scrape_pages(urls=[...], mode="fast")   # <= 10 URLs
pages = [r for r in batch["results"] if "error" not in r]
failed = [(r["url"], r["error"]) for r in batch["results"] if "error" in r]

# Raw HTML when you need to parse structure yourself
res = scrape_page(url="https://example.com", extraction="html")
from bs4 import BeautifulSoup
soup = BeautifulSoup(res["content"], "html.parser")
titles = [h1.get_text() for h1 in soup.find_all("h1")]

Batches run concurrently — 8 at a time in fast mode, 2 at a time in browser / stealth (browser sessions are memory-heavy). More than 10 URLs means more than one call.

Direct Python API (Advanced)

For selectors, sessions, spiders, or when you need the full Page object. Requires imports. Async.

Fetcher (Fast HTTP — Tier 1)

pythonCopy code
from scrapling.fetchers import AsyncFetcher

page = await AsyncFetcher.get("https://example.com", stealthy_headers=True)
print(page.status)       # 200
print(page.body)         # Raw bytes
print(page.headers)      # Response headers

# CSS selectors (Scrapy-style pseudo-elements)
titles = page.css("h1::text").getall()
links = page.css("a::attr(href)").getall()

# XPath
items = page.xpath("//div[@class='item']/text()").getall()

# BeautifulSoup-style
divs = page.find_all("div", class_="content")

DynamicFetcher (Browser — Tier 2)

pythonCopy code
from scrapling.fetchers import DynamicFetcher

page = await DynamicFetcher.async_fetch(
    "https://spa-website.com",
    headless=True,
    network_idle=True,
    disable_resources=True,
    timeout=30000,          # milliseconds here, unlike the MCP tools
    wait_selector=".data-table",
)
rows = page.css("table.data-table tr")
for row in rows:
    cells = row.css("td::text").getall()

StealthyFetcher (Anti-Bot — Tier 3)

pythonCopy code
from scrapling.fetchers import StealthyFetcher

page = await StealthyFetcher.async_fetch(
    "https://protected-site.com",
    headless=True,
    solve_cloudflare=True,
    network_idle=True,
)

Sessions (Persistent Connections)

pythonCopy code
from scrapling.fetchers import FetcherSession

with FetcherSession(impersonate="chrome") as session:
    login_page = session.post("https://site.com/login", data={...})
    dashboard = session.get("https://site.com/dashboard")
    data = dashboard.css(".user-data::text").getall()

Spider (Multi-Page Crawl)

pythonCopy code
from scrapling.spiders import Spider, Request, Response

class PriceScraper(Spider):
    name = "prices"
    start_urls = ["https://example.com/products"]
    concurrent_requests = 5

    async def parse(self, response: Response):
        for product in response.css(".product"):
            yield {
                "name": product.css(".name::text").get(),
                "price": product.css(".price::text").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield Request(next_page)

spider = PriceScraper()
result = spider.start()
result.items.to_json("work/<task_name>/data/prices.json")

Converting HTML to Markdown

Only needed when you fetched HTML yourself — scrape_page(extraction="markdown") already does this.

pythonCopy code
import html_to_markdown

markdown = html_to_markdown.convert(
    html_string, html_to_markdown.ConversionOptions(extract_metadata=False)
).content

# Article-only extraction (strips nav/ads/boilerplate)
import trafilatura

article = trafilatura.extract(html_string, output_format="markdown", favor_recall=True)

When to Use Which