Driving The Tui

Driving the live TUI

The TUI exposes a loopback control API so an assistant can operate it and read what a user would see. Use it. Never sleep.

Isolate memory before you start it — every time

Set GAIA_MEMORY_DB to a throwaway file in every drive. The agent behind the TUI writes to the user's real ~/.gaia/memory.db by default, and anything you say while driving becomes a permanent fact about the user:

bashCopy code
export GAIA_MEMORY_DB=/tmp/gaia-drive/memory.db     # delete between runs

A drive once planted a persona's overdue deadline; days later the user said "sweet!" and got "Priya needs that Fernbrook deck ASAP." back. The false bug reports this skill exists to prevent have a mirror image — a real report caused by a test.

gaia eval agent already resets memory between scenarios; a hand-driven session has no such cleanup, so isolation has to come from the environment. A blank value, or one naming a directory, is a startup error rather than a fall back to the real store — if the agent refuses to start, fix the path, don't unset the variable. GAIA_HOME selects $GAIA_HOME/memory.db when GAIA_MEMORY_DB is unset; it does not relocate config, logs, or every other ~/.gaia path. Config uses GAIA_CONFIG_DIR. Use a separate OS user or container for complete isolation.

Run it beside the user's own session, not through it

The daemon is machine-wide and the sidecar is shared. A TUI launched in dev mode hands that daemon its own source directory, so driving a checkout can swap the agent out from under a session the user is in the middle of using. Give the drive its own daemon instead:

bashCopy code
export GAIA_DAEMON_HOME=/tmp/gaia-drive/daemon   # own daemon + own sidecar
export GAIA_TUI_HOME=/tmp/gaia-drive/tui         # own control.json + token
export GAIA_MEMORY_DB=/tmp/gaia-drive/memory.db  # see above

With those set, ps aux | grep gaia-tui before and after should show the user's processes untouched. The readiness gate still needs gaia-agent on PATH; a one-line shim that execs python -m gaia_agent.server from the checkout under test is enough, and is how you pin which source the drive exercises.

Start it

bashCopy code
/path/to/gaia --control-port 8815     # --control also works (auto-assigns)

Off by default: --control is false, binds 127.0.0.1 only (control.Host), and is bearer-token authenticated (token in ~/.gaia/tui/control.json, mode 0600, compared with subtle.ConstantTimeCompare). Verified unreachable from the host's LAN address. Launch it in its own Terminal window — a long-lived process started from a background tool call gets SIGKILLed (exit 137).

Launch it in Windows Terminal, never bare cmd.exe

A cmd.exe window spawned by Start-Process lands in legacy conhost, which reports no colour support — and the TUI then renders with no colour and no syntax highlighting at all. It is not a rendering bug and it is not the capture stripping ANSI; the process genuinely emits zero escape sequences.

The chain: theme.Init() and components.PrimeRenderer() resolve the palette once at startup (prepareTerminal, internal/ui/app.go). detectStyle() (components/markdown.go) returns styles.NoTTYStyle when the terminal reports no colour, glamour then drops every chroma token, and answerPanelStyle paints each code line one flat grey. Everything looks right except that code blocks are monochrome — which reads as "syntax highlighting is broken".

Launch through wt.exe so the console is Windows Terminal (truecolor):

bashCopy code
powershell.exe -NoProfile -Command \
  "Start-Process wt.exe -ArgumentList @('new-tab','--title','GAIA','cmd.exe','/k','<abs path>\run.bat')"

Two traps in that line: a --title containing a space breaks wt's own argument parsing (error 0x80070002), so keep it one word; and put the env vars (GAIA_TUI_HOME, PYTHONPATH, GAIA_AGENT_LOG) inside the .bat, because a new tab handed to an already-running Windows Terminal inherits that process's environment, not your shell's.

To check colour rather than guess: GET /control/v1/screen?format=ansi and count \x1b. Zero on a frame that should be styled means the profile degraded — relaunch under wt.exe.

Endpoints

/control/v1/ — status · screen · keys · mouse · text · wait · frames · recording · resize

view is one of splash · preflight · chat · unknown — there is no hub screen to navigate, since gaia-tui boots straight into one agent's chat behind a readiness gate. On preflight, blocker names the row refusing the launch (binary, lemonade, model, daemon, sidecar, mailbox); wait on {"state": {"view": "chat"}} rather than a screen substring. esc quits on preflight (no screen behind it) unless the d details pane is open, which it closes first, and clears the composer on an idle chat.

Proof: screenshots and video

screencapture is blocked on this machine, and a plain-text dump cannot show colour, alignment, or which cell a link sits in. The control API renders the frames itself instead.

bashCopy code
# a still — a real picture of the current frame
curl -sH "Authorization: Bearer $TOK" \
  "http://127.0.0.1:$PORT/control/v1/screen?format=svg" -o shot.svg

# the session so far, replayed at the speed it happened, looping
curl -sH "Authorization: Bearer $TOK" \
  "http://127.0.0.1:$PORT/control/v1/recording?limit=45" -o session.svg

Both return the SVG document itself, not JSON — write it straight to a file. SVG rather than PNG so there is no font to ship and no rasteriser to depend on; every box-drawing glyph and emoji renders because the viewer draws it.

There is no start/stop for recording: the state keeps every frame it draws with the millisecond it was drawn, so the interesting moment is already captured by the time you realise it was interesting. limit picks how many of the most recent frames to include (the ring holds 200). since skips ahead — take latest_seq from GET frames before an action, then pass it as since afterwards to get a clip of just that action.

Send them with SendUserFile. A still is ~8KB; a 45-frame clip is ~270KB.

Sharing a capture

A still goes straight into a conversation. A video usually cannot: connectors that reach Drive take file content inlined, so a 600KB clip is ~800KB of base64 through the model. tui/scripts/share-capture.sh uploads with rclone instead and prints a link:

bashCopy code
tui/scripts/share-capture.sh session.mp4 shot.jpg

One-time setup belongs to the person whose Drive it is — it opens a browser for them to grant access, and the token lands in their rclone config rather than in a transcript:

bashCopy code
rclone config create gaia-drive drive scope=drive

Override the destination with GAIA_CAPTURE_REMOTE / GAIA_CAPTURE_FOLDER.

Turning stills into a video

format=svg is exact and needs no font, but not every viewer plays an animated SVG, and macOS qlmanage is not a usable rasteriser here — it stretches the document into a square and ignores preserveAspectRatio, which silently cuts the header and status bar off every frame. Render in a browser, which honours the document's own size:

bashCopy code
curl -sH "Authorization: Bearer $TOK" "$BASE/control/v1/frames?limit=200" \
  | gaia-tui-capture stills -out ./frames     # one SVG per frame, styling kept
# rasterise (browser canvas, or rsvg-convert on Linux), then:
gaia-tui-capture gif -in ./frames -out session.gif

For H.264 rather than GIF, AVFoundation will encode a PNG sequence on macOS and ffmpeg -framerate 3 -i f%04d.png out.mp4 anywhere it is installed.

Mouse

bashCopy code
curl -sX POST -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"events":[{"action":"wheel_up","x":40,"y":10}]}' \
  "http://127.0.0.1:$PORT/control/v1/mouse"

Actions: wheel_up · wheel_down · click · double_click · right_click · move. Coordinates are absolute screen cells in the same frame /screen reports, so the way to click something is to read the screen, find the row and column, and click that:

bashCopy code
curl -sH "Authorization: Bearer $TOK" "…/screen?format=plain" | python3 -c "
import sys,json,re
for i,l in enumerate(json.load(sys.stdin)['screen'].splitlines()):
    m = re.search(r'https://\S+', l)
    if m: print(i, (m.start()+m.end())//2); break"

A double click is two presses, not a distinct event — that is what a terminal sends and what the app has to recognise.

Injection bypasses the terminal's mouse gate. Events go straight into the program's message queue, so they arrive even when the app has released the mouse and a real user's click would reach nothing. The endpoint refuses with mouse_not_captured while state.chat.mouse_owner is terminal, precisely so this cannot report a pass for a gesture nobody can make. The terminal owns the mouse by default (so drag-select works), which means clicks and wheel ticks need keys ctrl+t first; scroll with up/down keys otherwise — that is what the wheel sends under alternate scroll mode. If you need to prove what the terminal is doing, read the escape sequences off the pty (below).

Diagnostics: state.chat

GET status carries the transcript's own state, and it is what tells a real defect from a mis-aimed test:

jsonCopy code
{"scroll_y": 8, "content_rows": 56, "at_bottom": false, "follow_tail": false,
 "mouse_owner": "terminal", "select_mode": true,
 "viewport_rows": 33, "header_rows": 2, "help_open": false, "messages": 6}

• scroll_y / at_bottom / follow_tail — "I scrolled and nothing moved" is indistinguishable from "I was already at the top" without these. A scroll that registered moves scroll_y and clears follow_tail.
• content_rows vs viewport_rows — when content fits the window there is nothing to scroll, and a scroll test there passes vacuously. Check this before concluding the wheel works; fill the transcript first if it does not.
• mouse_owner / select_mode — "my click did nothing" versus "the app never had the mouse".
• header_rows — how many screen rows sit above the transcript. Add it to a content row to get a screen row to click.

What the control API cannot see: the pty

Whether the app asked the terminal for the mouse is not model state — it is bytes on the wire. Run the TUI under a pty with a real size and read them:

pythonCopy code
pid, fd = pty.fork()                      # child: os.execvpe(...)
fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 40, 130, 0, 0))

A script-spawned pty has size 0x0, Bubble Tea then lays nothing out, and resize refuses to grow past it — which reads as "the TUI is broken" when it is the harness. Then grep the captured stream:

Decode an OSC 52 payload to prove what was copied, not merely that something was. A click that should open a browser is best checked by putting a stub open (or xdg-open) earlier on PATH that appends its argument to a file — you get the exact URL, and no browser windows.

Logs, traces, and errors

• GAIA_AGENT_LOG=<path> — the agent's own log for the session.
• --trace=<path.jsonl> — what the agent actually did, step by step.
• --dev — puts the model chip, step count and the per-turn footnote (elapsed · ttft · tokens · tok/s · steps) on screen, and routes agent logging to DEBUG.
• A failed action surfaces as a status line in the transcript rather than a crash, so screen?format=plain is where an error shows up. Read it before concluding a keystroke did nothing.

The rule: wait, don't sleep

POST /control/v1/wait blocks server-side until a condition holds: {"contains": "..."}, {"absent": "..."}, or {"state": {...}} (ANDed), with timeout_ms. A turn takes 12-90s depending on the model and tools; a fixed sleep is either a wasted minute or a half-rendered capture. Waiting on absent: "streaming" returns the instant the turn ends.

/tmp/drive.sh (recreate if missing — /tmp is cleared) should expose: keys · text · resize · wait <s> · gone <s> · ask <text> · screen [lo hi], where ask types, submits, and blocks on gone streaming.

Capture mistakes that cause false bug reports

• Always screen 0 999. A cropped capture once produced a fabricated "uninstall silently does nothing" — the status line is the second-to-last row.
• resize FIRST or cols/rows are 0 and everything wraps wrongly.
• Look at the picture, not just the text. Render screen?format=svg in a browser and screenshot it; a doubled cursor, a leaked label, or a repeated line is invisible in a grep. Filter cursors to the focused input's row: it must never hold more than one entry, and that entry must have no bar glyph (┃, │) in its left. Zero entries is the cursor's blink-off phase, so re-sample before calling it missing.
• format=plain strips ANSI — but format=ansi does not, and it is the fastest way to settle a colour question. Count \x1b in the result: a styled frame returns dozens (a healthy header is \x1b[1;38;2;181;224;141mGAIA), and zero means the colour profile degraded at launch — see the Windows Terminal note above, not a renderer bug.
• Check the screen you are actually on before sending keys. Keys sent to the wrong screen do nothing and read as "the binding is broken".
• A test that cannot fail is not a pass. Scrolling a transcript that fits the window, clicking where no link is, waiting on a condition already true — each returns success having proved nothing. Assert the precondition first (content_rows > viewport_rows), and where you can, run the negative case too: click one column off the link and confirm nothing opens.
• Prove the gate, not just the happy path. A click that works is half the story; the other half is that it does not work where it must not — under an open /help panel, or on prose. Both of those were real defects here, and both passed every unit test before a live drive caught them.
• screencapture is blocked on this machine. Use screen?format=svg.

Know which build you are driving

Stacked branches are siblings; no single branch contains everything. A leaf build shows behaviour already fixed elsewhere — this produced a bug report for something fixed on another branch. Build from a merged integration branch, or say explicitly which slice you tested.

Similarly, mode: user runs the published frozen sidecar, which is routinely older than source (2.4 vs 2.6). When testing an agent from a checkout, set the same mode on the TUI launch so its ensure request agrees with the sidecar:

bashCopy code
GAIA_EMAIL_AGENT_MODE=dev /path/to/gaia-tui --control-port 8815

Use GAIA_GAIA_AGENT_MODE=dev for the flagship gaia agent. The TUI resolves the caller checkout and sends the per-agent source directory to the daemon; starting a dev sidecar separately while the TUI still defaults to user creates a mode conflict. Check api_version in GET /daemon/v1/agents.