Testing The Gaia Agent

Testing the flagship GAIA agent through the TUI

Companion to driving-the-tui (which covers the control API mechanics). This one covers what to test and how to know it actually worked — written from a full session of driving the live agent, including every trap that cost an hour.

The ladder tests capabilities. Users have conversations.

Read this before trusting a green ladder. Every rung below is a self-contained prompt — it names its repo, its numbers, its subject. So the ladder ran green for a whole day while the TUI agent had no conversation history at all: every turn reached the model as system prompt + current question, nothing else. The bug surfaced the moment a real user typed a follow-up:

triage amd/gaia → three issues listed "cool, can you print issue 2975?" → "I need to know which repository it belongs to"

Three things hid it, and all three are worth knowing:

1. L3 looks like proof of continuity and is not. "What is my favourite colour?" passes across turns via the persistent memory store, a different mechanism entirely. Its green tick actively masked the gap.
2. The stdio test asserting turn-to-turn state passes for the wrong reason. test_the_agent_survives_between_turns asserts OBJECT state (agent.loaded_skills) survives — it does, the agent is the same object. History is not accumulated object state; nobody was appending to it.
3. The HTTP surface populates history, so any test at that layer passes. The defect was transport-specific, and only the TUI used the broken transport.

So always finish with a follow-up that cannot stand alone. Use a pronoun or a bare number and give it nothing else:

The second pair is the cheap canary — two short turns, no tools, no network. Run it first. If it answers "no fruit has been mentioned", stop: history is broken and every other result is measuring an agent with amnesia.

The one rule

A plausible answer is not a passing test. The flagship's worst failure mode is answering confidently when its tools are missing. It once produced a polished "here's how I'd triage that" paragraph while having zero GitHub tools registered. Every capability claim must be checked against ground truth from outside the agent:

bashCopy code
# agent said: #2958, #2955, #2953
gh issue list --repo amd/gaia --limit 3 --json number,title   # must match exactly

If you cannot independently verify a result, report it as unverified. Say so plainly.

Rules about the machine — ignore these and you will measure noise

Each of these cost real hours in the session this skill came from, and each produces symptoms that look like product bugs.

1. Exactly ONE TUI at a time

Kill every existing instance before launching, and never leave a second one running:

bashCopy code
# Windows
for p in $(tasklist //FI "IMAGENAME eq gaia-drive.exe" //FO CSV //NH | cut -d, -f2 | tr -d '"'); do
  taskkill //PID $p //F
done

Two TUIs is not merely wasteful:

• They overwrite each other's ~/.gaia/tui/control.json — same pid/port/token file — so your driver silently attaches to whichever launched last. A query you never sent appears in your transcript; keys you send land in someone else's session. This happened in both directions in one day, and each time looked like a TUI bug.
• Each spawns its own agent child, so they compete for the model and every turn slows.
• The user is memory-constrained; two instances is a real cost, not a rounding error.

GAIA_TUI_HOME isolates the discovery file so concurrent agents stop hijacking each other — it does not remove the model contention. One TUI, always.

2. Never run an eval while testing the agent

gaia eval agent and the TUI both drive the single-slot Lemonade backend. Running them together makes every turn 2-5x slower and the slowdown reads as "the agent is extremely slow" — a product complaint caused entirely by the harness. Measured on the same box, same build:

Worse, CLAUDE.md warns that concurrent runs race the model slot and can produce chaotic, meaningless failures (BLOCKED_BY_ARCHITECTURE, INFRA_ERROR, ctx-size errors) that get mistaken for regressions.

Check before you start, and check again when things feel slow:

bashCopy code
powershell.exe -NoProfile -Command "Get-CimInstance Win32_Process -Filter \"Name='python.exe'\" | Select-Object -ExpandProperty CommandLine" | grep -iE "eval agent|ui.server"

Evals are a pre-merge gate, not a testing-session activity. When a change requires one (CLAUDE.md lists the surfaces — prompts, tool schemas, tool-call parsing), record it as outstanding and run it when the box is quiet and nobody is driving the TUI. Never run two evals at once, either.

3. Give your session a private agent log

Every agent appends to ~/.gaia/logs/gaia-agent.log. When anything else is running an agent — a parallel task, a second harness — the file interleaves, and a neighbour's tool timeout reads as your session's failure. Set GAIA_AGENT_LOG in the launcher:

powershellCopy code
$env:GAIA_AGENT_LOG = 'C:\...\gaia-tui-test\logs\agent-session.log'

Lines also carry pid:NNNN, so the shared default is still attributable when you forget. This is not hypothetical: a 180s run_shell_command timeout was nearly filed as a shell bug here before the record turned out to belong to another process. Confirm the pid in the log matches the gaia-agent.exe your TUI spawned before believing anything.

4. Point memory at a throwaway DB — ALWAYS

GAIA_MEMORY_DB is mandatory in every launcher. Without it the agent writes to the user's real ~/.gaia/memory.db, and everything you plant during a test drive becomes a permanent fact about the user:

powershellCopy code
$env:GAIA_MEMORY_DB = 'C:\...\gaia-tui-test\memory\test-memory.db'

This is the eval-runner rule applied to interactive sessions. gaia eval agent already resets state between memory scenarios (GAIA_MEMORY_ADMIN=1 + memory_clear(scope=all) in src/gaia/eval/runner.py); a TUI test drive has no such cleanup, so isolation has to come from the environment.

It is not hypothetical. A ladder run planted a persona's overdue deadline; days later a real session answered the user's "sweet!" with "Priya needs that Fernbrook deck ASAP." The user's second brain had been quietly seeded by a test.

Delete the file between runs to test the cold-start path — a warm store hides first-run bugs the same way a warm model cache hid #1655.

A bad value is fatal on purpose. GAIA_MEMORY_DB set to a directory, or set blank, raises at startup rather than falling back to the real store — a harness that believes it is isolated but is not is the whole failure mode. If the agent will not start, read the error; do not unset the variable.

GAIA_HOME selects $GAIA_HOME/memory.db when GAIA_MEMORY_DB is unset. It does not isolate the whole ~/.gaia tree: config uses GAIA_CONFIG_DIR, and logs and other state may still use the real home directory. Use a separate OS user or container when the harness needs complete isolation.

Which surface you are testing

One binary, two surfaces — always state which:

Launching bare and typing lands your text in the Hub's filter box, not a chat composer. That produced a fake bug report once.

Setup

1. Build

bashCopy code
cd tui && go build -o bin/gaia-drive.exe ./cmd/gaia

Do not launch while a build is writing the binary — the file lock makes the launch silently fail. Build, then launch.

Go must be on PATH (export PATH="/c/Program Files/Go/bin:$PATH" on Windows). Note that gofmt -l flags nearly every file on a Windows checkout — that is CRLF, not real formatting drift. Check gofmt -d <file> | cat -A for ^M before you "fix" anything.

1b. The agent binary the TUI spawns

The TUI launches gaia-agent from PATH (catalog.go, BinaryPath). A source checkout does not have it — the console script only exists once the hub package is installed:

bashCopy code
uv pip install --python <venv>/Scripts/python.exe \
  -e hub/agents/gaia/python -e hub/agents/chat/python --no-deps

--no-deps is mandatory: without it pip pulls amd-gaia from PyPI and the agent imports THAT instead of your worktree. Put the venv's Scripts/ on PATH in the launcher or the TUI cannot find gaia-agent.

1c. The flagship ships with NO skills

gaia_agent/skills/ holds only .gitkeep, nothing stages hub/skills/ into it, and every skills: / skill_sets: / default_skill_set: key in gaia-agent.yaml is commented out. So L5–L7 cannot pass on a clean checkout — not because the agent is broken, but because it has nothing to load.

Install the one you are testing, and copy it rather than gaia skill import — import re-stamps the tier experimental, which is not what ships, and refuses a skill whose grants (like shell:execute:gh) sit above that tier's ceiling:

bashCopy code
cp -r hub/skills/github-triage ~/.gaia/skills/
gaia skill list      # expect: github-triage  2.1.0  community  user

Also note gh is refused until the skill that grants it is loaded — the grant is shell:execute:gh. Asking for gh first produces a confident refusal that looks like a missing-tool bug and is not one.

2. Launcher (adapt paths, keep the structure)

Create launch-tui.ps1. Every line matters:

powershellCopy code
$root = '<ABSOLUTE PATH TO YOUR WORKTREE>'
$env:PYTHONPATH = "$root\src;$root\hub\agents\chat\python;$root\hub\agents\gaia\python"
$env:GAIA_TUI_HOME = '<A PRIVATE TEMP DIR — NOT ~/.gaia/tui>'
$env:GAIA_MEMORY_DB = '<A PRIVATE TEMP FILE — NOT ~/.gaia/memory.db>'
$env:GAIA_AGENT_LOG = '<A PRIVATE TEMP FILE — NOT ~/.gaia/logs/gaia-agent.log>'
$env:PYTHONIOENCODING = 'utf-8'
$inner = "cd /d `"$root`" && tui\bin\gaia-drive.exe run gaia --control-port 8817"
Start-Process -FilePath 'cmd.exe' -ArgumentList '/k', $inner -WindowStyle Normal

Launch with:

bashCopy code
powershell.exe -NoProfile -ExecutionPolicy Bypass -File <path>\launch-tui.ps1

PYTHONPATH is mandatory. An editable install can resolve gaia to a different worktree, and the agent then dies at import with ModuleNotFoundError: No module named 'gaia.ui.sse_translation'. Verify:

bashCopy code
python -c "import gaia; print(gaia.__file__)"   # must be YOUR worktree

GAIA_TUI_HOME is mandatory when other agents may be running — see machine rule 1 above. It gives you a private control.json (tui/internal/control/paths.go) instead of the shared ~/.gaia/tui/control.json that agents hijack from each other. It does not excuse running two TUIs.

GAIA_MEMORY_DB is mandatory always — see machine rule 4. Omit it and your test drive writes into the user's real second brain. Verify before you type anything:

bashCopy code
python -c "from gaia.agents.base.memory_store import resolve_memory_db_path as r; print(r())"

Do not use cmd //c start from Git Bash — MSYS mangles the arguments and no window opens. PowerShell Start-Process with a .ps1 avoids the quoting entirely.

3. Driver

Use util/tui_driver.py from the repo root (repoint CJ at your GAIA_TUI_HOME).

Why one process: process spawn costs 0.7–2.0s on a Windows/MSYS box with AV — curl --version alone measured 2051 ms. A bash driver spawning bash + 2 × python + curl per command cost ~4.8s per call. The control API itself is 3 ms. Batch every step of a test into ONE python process:

Copy code
5 control calls in one process: 15 ms total

Driving correctly

• Wait for streaming:true BEFORE waiting for streaming:false. Otherwise the idle-wait matches the pre-turn idle state and returns in 0.0s, and you will report a phantom instant answer.
• Press end before every capture or you capture stale scrollback and read an old turn as the current one.
• Never sleep to wait out a turn — poll status or use /control/v1/wait.
• Set PYTHONIOENCODING=utf-8 or captures die on cp1252 for the spinner glyphs.
• Do not resize larger than the real terminal — the control API returns 409 resize_exceeds_terminal; a bigger size shreds the frame.

The capability ladder

Run in order. Stop and diagnose at the first failure — later rungs depend on earlier.

L6 is the regression canary for a bug where the skill vanished between turns. L7 is the real test: it fails silently by producing a confident non-answer.

Diagnosing L7 failure

If it deflects ("first configure the connector…") it has no tools. Check, in order:

bashCopy code
# 1. Does it think it has tools?  (a NONE here is the smoking gun)
#    ask in the TUI: "List the exact names of every tool you can call that talks
#    to GitHub. If you have none, say NONE."

# 2. What did the loader actually register?
grep -E "Loaded skill|registered_tools" ~/.gaia/logs/gaia-agent.log | tail -5
#    "0 tool(s), 1 connector requirement(s)" + 'registered_tools': [] == no tools

# 3. Is the skill the version you think?
grep -E "version:|shell:execute|mcp:connect" ~/.gaia/skills/github-triage/SKILL.md

The installed copy at ~/.gaia/skills/<name>/SKILL.md is what the agent reads, not the repo copy. After editing the repo skill, sync it or the agent runs the old one.

Verifying the permission gate

The gh grant has three tiers, and the bug you are hunting is a command in the wrong one. Run the gate check — instant, no LLM, no TUI:

bashCopy code
python util/tui_driver.py gate      # prints each case, its tier, and ok/WRONG

Expect 13/13 as expected. What each tier means:

The two failures worth naming, because each looks fine on a green ladder:

1. A write silently landing in ALLOW. It ran and nobody was asked. The gate check catches it; a TUI session will not, because the write succeeding looks like the feature working.
2. An escalation landing in CONFIRM. Now gh auth token has a yes button. The whole point of keeping REFUSE separate is that a prompt the user learns to approve approves that too.

Then confirm end-to-end in the TUI, with the box quiet:

Answer n unless you actually want the issue filed. If you answer y, delete the issue afterwards — and note gh issue close is itself REFUSE, so that is a manual step on github.com.

Check the prompt with your eyes, not the event log

The event stream is not the screen, and only one of them is the product. A gate can emit a perfectly-formed needs_confirmation that the user never sees: the modal used to live inside the scrollable transcript, so a long enough session pushed it below the fold — and because a pending modal owns the keyboard, end and PgUp could not scroll to it either. Measured cost: 442s of ● GAIA streaming, no visible question, and the turn ended only because the tester pressed Esc. Every unit test passed the whole time.

So capture the frame and read it:

bashCopy code
python util/tui_driver.py screen      # the frame, as the terminal paints it

A prompt on the model but not in the frame is the same defect as no prompt at all — worse than a hard refusal, because a refusal at least ends the turn. tui/internal/ui/chat/confirmvisible_test.go asserts these against the rendered frame; add to it rather than to a test that only inspects m.confirmation.

Measuring streaming

Sample on-screen character count during a turn. Rising = streaming; one jump at the end = not.

Confound to avoid: total screen chars include scrollback, and a re-render can make the count drop. Scope the count to the current answer region (text after the last ▶ You: line), or scroll to a clean state first. A naive whole-screen count produced an unreadable series (1301 … 1437, 991) and proved nothing.

Robustness checks

Stress it with real and hostile files

Small synthetic prompts pass while users' files break things. Generate the stress corpus (110 MB, seeded, never committed) and ask about each file:

bashCopy code
python util/stress_corpus.py --out ~/Documents/stress --answers ~/stress-answers.json

Keep the key outside the corpus: a content search over the folder finds the key's copy of every answer and the test proves nothing.

A right answer is not a pass. Read the tool trail and the timings too. The needle in the 1,667-page PDF came back correct after 11 minutes: a tool timed out mid-index, the retry indexed it again, and the trail said "Indexed document (0 chunks)". Then check the backend, which the transcript never shows:

bashCopy code
# side requests that hit the token cap: a thinking model reasoning without end
grep "out=4096" <lemonade log>
# turns that re-read the whole conversation: the prompt cache was lost
grep "Inference completed" <lemonade log>    # compare in= across turns

Run each case on both default models. Qwen3.6 reasons before every answer and Gemma does not, so a bug often shows up on only one of them. In the Agent UI, answer every permission prompt within its countdown: a prompt that times out is a denial, and the turn that follows tests the denial path, not the feature.

Known-good baselines (Gemma-4-E4B, GPU, quiet box)

If everything is 2–5× slower, suspect the harness before the product — a stray eval or a second TUI, per the two machine rules above. Confirm the backend is actually up and on the right port:

bashCopy code
curl -s http://127.0.0.1:13305/api/v1/health    # note: 13305, NOT 8000

Lemonade has died on its own mid-session more than once. Check it before blaming a change. Restarting it is not lemonade-server serve — that binary may not exist, and lemonade.exe is the client and rejects serve:

bashCopy code
powershell.exe -NoProfile -Command "Start-Process 'C:\Users\<you>\AppData\Local\lemonade_server\bin\LemonadeServer.exe' -WindowStyle Minimized"
curl -s -X POST http://127.0.0.1:13305/api/v1/load -H "Content-Type: application/json" \
     -d '{"model_name":"Gemma-4-E4B-it-GGUF"}'      # pre-warm, or turn 1 pays ~3.5 min

A cold first turn is ~240s (ttft ~228s) while both the LLM and the embedding model load; warm turns are ~6s. Pre-warm before timing anything, or the first number is a model load and you will report it as agent latency.

When a shell command hangs for exactly 180s

Two real bugs produced this, both fixed — but the diagnostic pattern generalises to any tool the agent shells out to.

1. Check for orphans. Get-CimInstance Win32_Process -Filter "Name='gh.exe'". A live child whose parent is gone means subprocess.run killed the cmd.exe at its inner timeout, then blocked forever in a second communicate() on pipes the grandchild still holds. The 180s you see is the OUTER tool timeout.
2. Compare against the same command from a shell. 0.07s outside vs a hang inside means the environment the agent spawns into, not the command.
3. Suspect stdin first. capture_output redirects stdout/stderr and leaves stdin inherited — the agent's stdin is the TUI's pipe, open and never written. Anything that reads or probes it waits on input that cannot arrive.
4. Then suspect the decode. Bare text=True decodes with the OS locale codec (cp1252 on Windows) inside subprocess's reader thread. One unmappable byte kills that thread and run() returns returncode 0 with empty stdout — a success with the output silently discarded. gh issue list on amd/gaia hits it, because issue #2962's title contains "⚠️".

Both failures lie in the same direction: the agent reports a confident, wrong explanation ("a networking bottleneck", "no issues found") rather than an error. Always diff the agent's answer against gh directly.

Re-run the ladder after every merge — a merge can kill Enter

Merging four feature branches took the ladder from 7/7 to 0/7, and nothing anywhere reported an error. Every rung just timed out at 20s, because Enter had stopped submitting: one branch added a heuristic treating an Enter within 50ms of the last keystroke as a pasted line break, and the control API delivers a line and its Enter back to back.

Two lessons, both cheap to act on:

• The ladder is the merge gate, not just the feature gate. go test ./... passed the whole time — the broken behaviour was covered by a test asserting the new intent. Only driving the real TUI caught it.
• A timing heuristic on input will find your harness. Anything of the form "too fast to be a person" is also true of the control API, and of a fast typist. Treat a change that infers intent from keystroke timing as a red flag.

The signature to recognise: tui_driver.py ladder returns rungs whose captured output is the startup banner rather than an answer, each taking exactly the idle-wait timeout. That means no turn ever started — look at input handling, not at the agent.

Watch out for a launcher that steals focus

Start-Process -WindowStyle Normal puts the new window in front, so keystrokes the user is typing elsewhere land in the TUI. That produced turns arriving as ▶ You: life Load the xlsx skill and ▶ You: tCan you reliably… — fragments of the user's own typing, which read convincingly as an input bug in the product.

It reproduced 2 of 4 launches and then 0 of 3. If you see junk prepended to a turn, check whether a human was typing before you write it up. A phantom bug filed against the product costs more than the hour spent disproving it.

Reporting

Per CLAUDE.md → How You Communicate: open with whether it works, in one plain sentence, then captures and detail beneath.

Specific to this skill:

• Paste real captured text, never paraphrase. A paraphrased frame hides the bug.
• State every rung you did not reach. An unstated gap reads as a pass.
• Verify before attributing a bug to your change. The tree often has other agents' uncommitted work — git status / git diff first. A "broken build" once turned out to be a stale test cache; a suspected regression turned out to be a rendering-only diff.
• Correct yourself out loud. A wrong bug report costs more than a missing one.