Web Scraping

Web Scraping Skill

Description

Extract data from web pages using JavaScript and CSS selectors. This skill uses real browser automation to navigate to pages and extract content using JavaScript execution. It can extract webpage titles using document.title and body text using document.body.textContent, providing actual content from live web pages.

Dependencies

None

Parameters

Return Value

Returns a SkillResult with the following data structure:

jsonCopy code
{
  "url": "string",
  "selector": "string",
  "attributes": ["string"],
  "data": "extracted content"
}

Usage Examples

Basic Text Extraction

kotlinCopy code
val result = registry.execute(
    skillId = "web-scraping",
    context = context,
    params = mapOf(
        "url" to "https://example.com",
        "selector" to ".content"
    )
)

Extract Multiple Attributes

kotlinCopy code
val result = registry.execute(
    skillId = "web-scraping",
    context = context,
    params = mapOf(
        "url" to "https://example.com/products",
        "selector" to "a.product-link",
        "attributes" to listOf("text", "href")
    )
)

Tool Call Specification

kotlinCopy code
ToolSpec(
    domain = "skill.debug.scraping",
    method = "extract",
    arguments = [
        "url: String",
        "selector: String",
        "attributes: List<String> = listOf('text')"
    ],
    returnType = "Map<String, Any>",
    description = "Extract data from a web page using CSS selectors"
)

Error Handling

The skill returns a failure result in the following cases:

• Missing required parameter url
• Missing required parameter selector
• Invalid URL format (must start with http:// or https://)

Lifecycle Hooks

onLoad

Initializes resources and loads configurations when the skill is registered.

onBeforeExecute

Validates URL format before execution. Returns false if URL doesn't start with http:// or https://.

onAfterExecute

Records the timestamp of successful scraping operations in shared resources.

validate

Validates skill configuration and environment. Always returns true for this skill.

Implementation Notes

• Headless mode is the default for AI agents: Always open browsers in headless mode unless the user explicitly requests a visible browser window (e.g., "show me the browser", "open visibly", "headed"). Headless mode is faster, uses fewer resources, and avoids unnecessary GUI windows. If user participation in the page interaction is required — e.g., logging in or entering a verification code (CAPTCHA) — open a headed browser so the user can see and act on the page. Also open headed as a one-shot escalation when the site blocks the headless browser as a bot (CAPTCHA/challenge/interstitial page, "unusual traffic", or an empty body where content was expected): close, then open --headed <url> with the same session and retry once, telling the user that anti-bot protection forced the switch. If the headed retry is blocked too, stop and report instead of looping.
• Uses JavaScript execution via WebDriver.evaluate() to extract real webpage content
• Extracts document title using document.title
• Extracts page text using document.body.textContent
• Falls back to PulsarSession-based extraction if WebDriver is not directly available
• Returns simulated data only if neither WebDriver nor PulsarSession is available in the skill context
• Supports both synchronous and asynchronous execution patterns
• Thread-safe and can be used in concurrent environments
• Text content is limited to 5000 characters to avoid overwhelming responses

See Also

• Form Filling Skill
• Data Validation Skill