Browser Jev Automation

Browser Jev Automation

Zero-dependency browser harness. Programs read the DOM and execute actions, Jev makes every judgment call, an LLM writes text only when the goal does not already contain the value. Built in-house; no pip or npm packages.

Quick start

bashCopy code
python3 scripts/jev-browser-agent.py \
  --url http://127.0.0.1:8000/ \
  --goal "Sign in with username alice, choose country Canada, and submit." \
  --check-text-contains "Welcome alice" --json-compact

Requires TYPESAFE_API_KEY, Node 22+, and a local Chromium (Playwright cache or CHROME_PATH). Output is one JSON object: status (done|blocked|budget|error), reason, steps, requests, final_url, verify, log (full probability distributions per step), usage.

Measured on a local login form: 5 steps, 7 Jev calls, 1.4 s wall, zero LLM tokens, goal verified.

Architecture: three tiers applied to browser control

One Jev call per decision cycle. Speculative fan-out: operation Choice + per-operation target Choices evaluated in one forward pass. Only the target matching the selected operation executes.

Components

CLI

Standalone page checks and audits

jev-browser-verify.py also opens pages itself:

bashCopy code
python3 scripts/jev-browser-verify.py --url http://127.0.0.1:8002/five-star \
  --goal "at least 8 promotions can be toggled" --goal "there is a Start button"
python3 scripts/jev-browser-verify.py --audit-file pages.json   # {"base": "...", "pages": {"/path": ["goal", ...]}}

One browser per audit, one snapshot per page, one Jev call per goal (~100-200 ms each). passed needs goal_met and evidence score >= 0.5. Same --allow-host, --header, --check-* flags as the agent.

Safety invariants

• Model output never becomes selectors, coordinates, or JavaScript. Every action carries an integer node id issued by snapshot.js; the driver resolves geometry itself and rejects covered, hidden, disabled, or read-only targets.
• The node registry lives in a CDP isolated world. Page scripts cannot see or rewrite it (live test asserts typeof window.__jevBrowser === "undefined" from the page).
• Loopback URLs only by default. --allow-host widens per host. If a click leaves the allowed origin (sign-in redirect, external link), the agent returns to the last good URL, tells Jev which action caused it, and blocks after three such trips.
• Secrets: the log shows (secret); after a secret is typed, every later snapshot is scrubbed before Jev, the text model, or the log sees it. Password inputs never expose their value ((filled)).
• Page text reaches Jev and the text model as untrusted data. Instructions say so explicitly. Every page string in a Jev state (title, URL, labels, values, history) is flattened to one capped line by scripts/lib/jev_untrusted.py, so it cannot forge a new element row. The text-model prompt wraps page data in <untrusted-page-data> tags after redaction and removes any copy of the tag the page planted. jev_router_common redacts every Jev payload before send.
• Jev answers are parsed strictly: an operation that was not offered, a target that was not offered, or a probability that is not a finite number in [0, 1] is a bad answer (RETRY, bounded by the stall guard), never executed.
• A CLICK on a link to a host outside loopback and --allow-host is refused before it is sent; a link with no known host is refused too. Text from the text model over 500 characters is refused, not typed.
• The text API needs https (loopback http allowed for tests) and never follows a redirect, so its key stays with the configured host. The Node driver and Chromium get an environment without keys, tokens, or the --secret-env and --header variables.
• jev-browser-verify.py --url and --audit-file do not send a page to Jev if it redirected off the allowed hosts.
• Chromium runs a throwaway profile. Sandboxed launch first; --no-sandbox only when Chromium reports "No usable sandbox" (user namespaces disabled). The ready line reports sandbox: true|false.

Key patterns (learned from jev-ultrafast)

• Speculative fan-out: one request asks "which operation?" and "which target for CLICK/TYPE_TEXT/SELECT?" at once. Unused heads are discarded.
• One read per cycle: observe runs snapshot.js once. Jev sees that snapshot.
• Semantic freshness guards: act compares page_key + the target's guard (value, enabled state, position, nearby text) for targeted actions, or the page marker for scroll/wait. A stale page returns stale: true; the loop re-observes without a stall penalty, capped at STALE_LIMIT (5).
• Decision consumed before mutation: the log entry is written before act, so a navigation cannot erase the record.
• Independent verification: DONE is a claim. jev-browser-verify.py runs as a separate question set; a rejected DONE goes back into history so Jev re-decides with that evidence. Three rejected DONEs mean BLOCKED.
• Full distributions logged: every step records operation_probabilities and target_probabilities.
• Text reuse only on identical context: cached by (url, target, label, current value).
• Transient Jev errors retry as WAIT up to STALL_LIMIT (3); unavailable blocks at once.
• Low-confidence picks are held: a non-DONE operation under LOW_CONFIDENCE (0.45) is not executed; the loop re-observes and counts toward the stall guard.
• Option order: the element table is priority-sorted, and Jev leans toward the first option on some inputs. For a high-stakes target (submit, delete, pay), ask the target Choice twice in the same request, once with the options reversed, and act only when both orders agree. The top-to-second probability ratio is a cheaper check on a close pick; measure it against LOW_CONFIDENCE before replacing it.
• Pre-filter options: offer only controls that are visible, enabled, and legal for the operation. Offered everything, Jev picks degenerate options repeatedly. A runner-up from probabilities is the retry after a no-effect action; never sample.
• Settle before judging: observe waits for the DOM to be quiet (600 ms, capped at 3 s) and reports quiet; WAIT pauses 1.5 s then settles up to 8 s; a rejected DONE triggers a WAIT before the next decision. Jev sees page.settled and history notes when content was still changing.
• Whole-page element table: rendered controls below the fold are offered too, marked offscreen; the driver scrolls them into view. Covered controls (modal backdrop, sticky header) are dropped at snapshot time, so an open modal leaves only its own controls.
• Links say where they go: navigates away to /path or leaves this site, and decide's rules forbid them unless the goal names that page.

Text value selection

Order for a TYPE_TEXT target:

1. --secret-env match on the field label.
2. Program extracts candidate literals from the goal (quoted strings, emails, username X, search for X, numbers).
3. Jev Choice over candidates plus NONE. Accepted at confidence >= 0.5.
4. LLM via Anthropic Messages API (urllib, no SDK). Strict {"text": "..."} reply.

The text model writes one value. It never picks actions or judges progress.

Screenshots and Box

When DOM text cannot identify the target (canvas, charts, icon-only controls, text in images), take a screenshot, ask a Box question for the target plus a "visible?" Noul, and click the box center. Keep deterministic verifiers as the veto. Load references/jev-images.md for the pattern and code; API and limits: images.md.

Phase 1: OBSERVE

observe evaluates snapshot.js in the isolated world. Output:

textCopy code
[1] heading  Sign in                          []
[2] textbox  Username                         [CLICK, TYPE_TEXT]
[3] combobox Country (value=USA)              [SELECT]   options 3:1 USA, 3:2 Canada
[4] checkbox Remember (checked=False)         [CLICK]
[5] button   Sign in                          [CLICK]

Plus actions (e2:fill, e3:sel:ca, scroll_down, wait), page_key, guards, marker, and page text (<= 6000 chars).

Gate: snapshot captured and scrubbed. Phase 2.

Phase 2: DECIDE

jev-browser-decide.py returns operation, target, confidence, needs_text, and both probability maps. Invalid targets resolve to BLOCKED. Jev unavailable resolves to BLOCKED with source: unavailable.

Gate: operation and target decided and logged. Phase 3.

Phase 3: EXECUTE

1. Resolve the observed action by (kind, index). Missing action: record, count toward stall.
2. TYPE_TEXT: pick text per the order above.
3. act with the snapshot's page_key, guards, marker. Stale: re-observe.
4. Driver settles: two animation frames or 50 ms; combobox fills wait up to 200 ms for visible options.

Gate: action executed. Phase 4.

Phase 4: VERIFY

• DONE: jev-browser-verify.py with deterministic checks. Verified: stop done. Rejected: history entry, re-decide.
• BLOCKED: stop.
• Stall guard: STALL_LIMIT (3) consecutive non-WAIT actions with an unchanged marker: stop blocked.
• Budget: MAX_STEPS 60, MAX_REQUESTS 120.

Gate: loop or stop. Every exit carries reason.

Walkthrough validity

• Count coverage without direct-URL fallbacks. A fallback navigates by URL when the player could not find the control, so it hides sections a player cannot reach (field case: 88 fallbacks on mobile against 43 on desktop, hiding 6 sections).
• Assert the landing route before you count a probe. A deep-link probe that lands on the wrong screen is inconclusive, not a negative.
• Check each tile's screen identity before you score it. When the layout has no nav rail (mobile), screen detection can fall back and score a tile as the wrong screen, which invalidates its scores; the screen under the viewport centre wins.
• Close open overlays (a command palette, a modal) before each click, and tag each failure as a harness failure or a product failure. Most click failures in one field walkthrough were the harness clicking behind an open overlay.

Long tasks: checkpoint search

The loop above picks every step with Jev. For tasks longer than a few steps, use checkpoint search: an LLM or the caller sets subgoals, and Jev beam-searches between them (../../meta/building-with-jev/references/composition-patterns.md, Checkpoint search; scripts/jev_search.py). Branching needs a way back to a kept state (re-navigate to the checkpoint URL and replay); count that cost. The agent loop does not run checkpoint search yet. Adopt it only after this eval:

• Run the same task set through all three arms, bucketed by task length (for example 1–5, 6–15, 16+ steps).
• Report per bucket: success rate (verified DONE), total tokens (Jev plus LLM), and wall time.
• Price and pace the eval per rule 7 of the production rules; use a held-out task set for the final report.

Request sizing and retries

Apply Jev production rules when you change the decide, text, or verify calls:

• Keep each request at or under the reliable size from python3 scripts/jev-size-probe.py --payload <dumped requests> (2.5–4k tokens via Gateway until measured). Large pages grow the element table: bound it by the priority sort and field limits rather than sending the whole page.
• Use Vercel AI Gateway. Too much context is the most common failure, and page snapshots are the usual cause: estimate each request before sending and trim the element table or split questions rather than send an oversized request. A ~100-token probe that returns proves the cause is size.
• Retry a lone fast Gateway 503 after 50–150 ms. Back off exponentially on 429/529, honoring Retry-After and retry-after-ms. Never retry 401/402/422.
• Every retry counts against --max-requests.

Error handling

Every Jev failure resolves to BLOCKED or a bounded WAIT retry. Scripts exit 0 with JSON. Preflight fails before Chromium launches on: remote URL without --allow-remote, Jev unavailable, unset secret variable, malformed --secret-env. A failed text tier blocks the run and names the cause in reason. Driver failures return status: error with the Node stderr tail.

Tests

Field-tested on a booking-sim web app (loopback dev server): closes the help modal, selects promotions, picks a mode, starts booking, spins the venue; a three-page audit script runs through --audit-file unchanged.

References

• references/jev-images.md: screenshot and Box locate pattern
• jev-ultrafast: patterns studied, code not reused
• docs/PHILOSOPHY.md: three execution tiers
• scripts/jev_router_common.py: validated_call_jev, bound_text, typesafe_available