Testdriver:Parse

Overview

Parse the screen with OmniParser v2 to find all visible UI elements. TestDriver returns structured data. This includes element types, text content, interactivity levels, and bounding box coordinates.

This method examines all of the screen and returns each element that it finds. Use it for these:

• To understand the full UI layout of a screen
• To find all clickable or interactive elements
• To build custom element logic
• To debug which elements TestDriver can find
• To do an accessibility audit

Syntax

javascriptCopy code
const result = await testdriver.parse()

Parameters

None.

Returns

Promise<ParseResult> - Object containing detected UI elements

ParseResult

ParsedElement

Examples

Get All Elements on Screen

javascriptCopy code
const result = await testdriver.parse();
console.log(`Found ${result.elements.length} elements`);

result.elements.forEach((el, i) => {
  console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
});

Find Clickable Elements

javascriptCopy code
const result = await testdriver.parse();

const clickable = result.elements.filter(e => e.interactivity === 'clickable');
console.log(`Found ${clickable.length} clickable elements`);

clickable.forEach(el => {
  console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
});

Find and Click an Element by Content

javascriptCopy code
const result = await testdriver.parse();

// Find a "Submit" button
const submitBtn = result.elements.find(e => 
  e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
);

if (submitBtn) {
  // Calculate center of the bounding box
  const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
  const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
  
  await testdriver.click({ x, y });
}

Filter by Element Type

javascriptCopy code
const result = await testdriver.parse();

// Get all text elements
const textElements = result.elements.filter(e => e.type === 'text');
textElements.forEach(e => console.log(`Text: "${e.content}"`));

// Get all icons
const icons = result.elements.filter(e => e.type === 'icon');
console.log(`Found ${icons.length} icons`);

// Get all buttons
const buttons = result.elements.filter(e => e.type === 'button');
console.log(`Found ${buttons.length} buttons`);

Build Custom Assertions

javascriptCopy code
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";

describe("Login Page", () => {
  it("should have expected form elements", async (context) => {
    const testdriver = TestDriver(context);
    
    await testdriver.provision.chrome({
      url: 'https://myapp.com/login',
    });

    const result = await testdriver.parse();
    
    // Assert expected elements exist
    const textContent = result.elements.map(e => e.content.toLowerCase());
    expect(textContent).toContain('email');
    expect(textContent).toContain('password');
    
    // Assert there are clickable elements
    const clickable = result.elements.filter(e => e.interactivity === 'clickable');
    expect(clickable.length).toBeGreaterThan(0);
  });
});

Use Bounding Box Coordinates

javascriptCopy code
const result = await testdriver.parse();

result.elements.forEach(el => {
  // Pixel coordinates
  console.log(`Element "${el.content}":`);
  console.log(`  bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
  console.log(`  size: ${el.boundingBox.width}x${el.boundingBox.height}`);
  console.log(`  position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
});

View Annotated Screenshot

javascriptCopy code
const result = await testdriver.parse();

// The annotated image shows all detected elements with bounding boxes
console.log('Annotated screenshot:', result.annotatedImageUrl);
console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);

How It Works

1. TestDriver captures a screenshot of the current screen
2. The image is sent to the TestDriver API
3. OmniParser v2 analyzes the image to detect all UI elements
4. Each element is classified by type (text, icon, button, etc.) and interactivity
5. Bounding box coordinates are returned in pixel coordinates matching the screen resolution

Best Practices

Copy code
```javascript
// Prefer this for clicking a specific element
await testdriver.find("Submit button").click();

// Use parse() for full UI analysis
const result = await testdriver.parse();
const allButtons = result.elements.filter(e => e.type === 'button');
```

Copy code
```javascript
const result = await testdriver.parse();
const interactive = result.elements.filter(e => e.interactivity === 'clickable');
const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
```

Copy code
```javascript
// Wait for page to stabilize
await testdriver.wait(2000);

// Then parse
const result = await testdriver.parse();
```

Copy code
```javascript
const result = await testdriver.parse();
console.log('View annotated screenshot:', result.annotatedImageUrl);
```

Related

• find() - AI-powered element location
• assert() - Make AI-powered assertions about screen state
• screenshot() - Capture screenshots
• Elements Reference - Complete Element API