Browser

Browser control (agent-browser + live takeover)

The sandbox runs a real, headful Chromium that the user watches live in the CubePlex browser panel and can take over at any time. You drive that same browser with the agent-browser CLI over CDP. Because the user sees exactly what you do, this is also how logins / OAuth / CAPTCHAs get solved: you navigate, and when a step needs a human you ask the user to take over in the panel.

First: start the browser stack, then attach (never launch your own)

bashCopy code
/usr/local/bin/start-browser.sh   # idempotent: brings up the live browser if not running
agent-browser connect 9222        # attach to that user-visible Chromium (CDP 127.0.0.1:9222)

start-browser.sh boots the headful Chromium the user watches. It is idempotent but not a blind no-op: if Neko/supervisord is up while Chromium died (user closed the last tab/window → empty desktop + CDP refused), it restarts Chromium and waits until CDP 9222 answers. Always run it first. Then agent-browser connect 9222 attaches to it. If connect still fails with "connection refused", run start-browser.sh again and wait a few seconds. Never start a fresh/headless browser — the user would see nothing and the session wouldn't be theirs.

Tell the user they can watch and take over in the CubePlex browser panel.

Keep the user's visual tab in sync with the agent's active tab

The user watches Chromium live through a streaming panel that CubePlex opens automatically when you start using the browser. A sandbox wrapper brings the session's current page to the foreground after every agent-browser command, so the live view follows the tab you are operating on.

Still switch explicitly when you create or choose a tab — that is how you pick which page the wrapper will focus:

bashCopy code
# After connect, list tabs and switch to the one you're about to work on:
agent-browser tab               # list all tabs
agent-browser tab t1            # make t1 the session tab (live view follows)

# After goto, verify the URL is what you expect:
agent-browser get url

# After creating a new tab, switch to it:
agent-browser tab new https://example.com
agent-browser tab t2

If the user reports seeing a different tab, run agent-browser tab and agent-browser tab <id> to re-sync. When taking over from a previous session, always list tabs and focus the one you intend to work on.

Learn the commands from the CLI (don't guess)

bashCopy code
agent-browser skills get core        # workflows + common patterns
agent-browser skills get core --full # full command reference

The CLI serves usage that matches the installed version. Common ones:

bashCopy code
agent-browser goto https://example.com   # navigate the visible page
agent-browser snapshot                    # accessibility tree with @eN element refs
agent-browser click @e12                  # click an element by its ref
agent-browser type @e8 "search text"      # type into a field
agent-browser get text|url|title          # read page state
agent-browser screenshot out.png          # save a screenshot to /workspace

Target elements by the @eN refs from snapshot (reliable), not by guessing selectors. Re-snapshot after the page changes.

When a step needs the human (login / OAuth / 2FA / CAPTCHA)

1. Navigate as far as you can (e.g. to the login page).
2. Tell the user plainly: "Please take over in the browser panel to log in / solve this, then tell me to continue." The user clicks Take over and acts.
3. Wait for the user to confirm, then resume — the session (cookies, localStorage) persists because it's the same browser, so you stay logged in for later steps.

Do not try to type passwords or solve CAPTCHAs yourself; hand those to the user.

Don't

• Don't detach or close the browser when finishing a task — leave it running for the user (the panel and the next task reuse it).
• Don't fall back to curl/HTTP fetch for pages that need JS or a login — use the browser.
• Don't let the user stare at a blank/stale tab while you're operating on a different one — always sync the visual tab.