Web Scraper

Web Scraper

Fetch web page content (text + images) and save as HTML or Markdown locally.

Minimal dependencies: Only requires requests and beautifulsoup4 - no browser automation.

Default behavior: Downloads images to local images/ directory automatically.

Quick start

Single page

bashCopy code
{baseDir}/scripts/scrape.py --url "https://example.com" --format html --output /tmp/page.html
{baseDir}/scripts/scrape.py --url "https://example.com" --format md --output /tmp/page.md

Recursive (follow links)

bashCopy code
{baseDir}/scripts/scrape.py --url "https://docs.example.com" --format md --recursive --max-depth 2 --output ~/Downloads/docs-archive

Setup

Requires Python 3.8+ and minimal dependencies:

bashCopy code
cd {baseDir}
pip install -r requirements.txt

Or install manually:

bashCopy code
pip install requests beautifulsoup4

Note: No browser or driver needed - uses pure HTTP requests.

Inputs to collect

Single page mode

• URL: The web page to scrape (required)
• Format: html or md (default: html)
• Output path: Where to save the file (default: current directory with auto-generated name)
• Images: Downloads images by default (use --no-download-images to disable)

Recursive mode (--recursive)

• URL: Starting point for recursive scraping
• Format: html or md
• Output directory: Where to save all scraped pages
• Max depth: How many levels deep to follow links (default: 2)
• Max pages: Maximum total pages to scrape (default: 50)
• Domain filter: Whether to stay within same domain (default: yes)
• Images: Downloads images by default

Conversation Flow

1. Ask user for the URL to scrape
2. Ask preferred output format (HTML or Markdown)
   • Note: Both formats include text and images by default
   • HTML: Preserves original structure with downloaded images
   • Markdown: Clean text format with downloaded images in images/ folder
3. For recursive mode: Ask max depth and max pages (optional, has sensible defaults)
4. Ask where to save (or suggest a default path like /tmp/ or ~/Downloads/)
5. Run the script and confirm success
6. Show the saved file/directory path

Examples

Single Page Scraping

Save as HTML

bashCopy code
{baseDir}/scripts/scrape.py --url "https://docs.openclaw.ai/start/quickstart" --format html --output ~/Downloads/openclaw-quickstart.html

Save as Markdown (with images, default)

bashCopy code
{baseDir}/scripts/scrape.py --url "https://en.wikipedia.org/wiki/Web_scraping" --format md --output ~/Documents/web-scraping.md

Result: Creates web-scraping.md + images/ folder with all downloaded images (text + images).

Without downloading images (optional)

bashCopy code
{baseDir}/scripts/scrape.py --url "https://example.com" --format md --no-download-images

Result: Only text + image URLs (not downloaded locally).

Auto-generate filename

bashCopy code
{baseDir}/scripts/scrape.py --url "https://example.com" --format html
# Saves to: example-com-{timestamp}.html

Recursive Scraping

Basic recursive crawl (depth 2, same domain, with images)

bashCopy code
{baseDir}/scripts/scrape.py --url "https://docs.example.com" --format md --recursive --output ~/Downloads/docs-archive

Output structure (text + images for all pages):

Copy code
docs-archive/
├── index.md
├── getting-started.md
├── api/
│   ├── authentication.md
│   └── endpoints.md
└── images/              # Shared images from all pages
    ├── logo.png
    └── diagram.svg

Deep crawl with custom limits

bashCopy code
{baseDir}/scripts/scrape.py \
  --url "https://blog.example.com" \
  --format html \
  --recursive \
  --max-depth 3 \
  --max-pages 100 \
  --output ~/Archives/blog-backup

Ignore robots.txt (use with caution)

bashCopy code
{baseDir}/scripts/scrape.py \
  --url "https://example.com" \
  --format md \
  --recursive \
  --no-respect-robots \
  --rate-limit 1.0

Faster scraping (reduced rate limit)

bashCopy code
{baseDir}/scripts/scrape.py \
  --url "https://yoursite.com" \
  --format md \
  --recursive \
  --rate-limit 0.2

Features

Single Page Mode

• HTML output: Preserves original page structure
  • ✅ Clean, readable HTML document
  • ✅ All images downloaded to images/ folder
  • ✅ Suitable for offline viewing
• Markdown output: Extracts clean text content
  • ✅ Auto-downloads images to local images/ directory (default)
  • ✅ Converts image URLs to relative paths
  • ✅ Clean, readable format for archiving
  • ✅ Fallback to original URLs if download fails
  • Use --no-download-images flag to keep original URLs only
• Simple and fast: Pure HTTP requests, no browser needed
• Auto filename: Generates safe filename from URL if not specified

Recursive Mode (--recursive)

• ✅ Intelligent link discovery: Automatically follows all links on crawled pages
• ✅ Depth control: --max-depth limits how many levels deep to crawl (default: 2)
• ✅ Page limit: --max-pages caps total pages to prevent runaway crawls (default: 50)
• ✅ Domain filtering: --same-domain keeps crawl within starting domain (default: on)
• ✅ robots.txt compliance: Respects site's crawling rules by default
• ✅ Rate limiting: --rate-limit adds delay between requests (default: 0.5s)
• ✅ Smart URL filtering: Skips images, scripts, CSS, and duplicate URLs
• ✅ Progress tracking: Real-time console output with success/fail/skip counts
• ✅ Organized output: Preserves URL structure in directory hierarchy
• ✅ Efficient crawling: Sequential with rate limiting to respect servers

Guardrails

Single Page Mode

• Respect robots.txt and site terms of service
• Some sites may block automated access; this tool uses standard HTTP requests
• Large pages with many images may take time to download

Recursive Mode

• Start small: Test with --max-depth 1 --max-pages 10 first
• Respect robots.txt: Default is on; only use --no-respect-robots for your own sites
• Rate limiting: Default 0.5s is polite; don't go below 0.2s for public sites
• Same domain: Strongly recommended to keep --same-domain enabled
• Monitor progress: Watch for high fail rates (may indicate blocking)
• Storage: Recursive crawls can generate many files; ensure sufficient disk space
• Legal: Ensure you have permission to crawl and archive the target site

Troubleshooting

• Connection errors: Check your internet connection and URL validity
• 403/blocked: Some sites block scrapers; the tool uses realistic User-Agent headers
• Timeout: Increase --timeout flag for slow-loading pages (value in seconds)
• Image download fails: Images will fall back to original URLs
• Missing images: Some sites use JavaScript to load images dynamically (not supported)