Gaia Testing

GAIA Testing

Test the way that catches regressions pytest misses: unit → integration → real-world, where the real-world tier drives the real interface on real hardware and returns screenshots, logs, traces, and timing as evidence. Features ship broken while CI is green — a RAG feature that worked in the backend but was hard-blocked in the UI, a release-note claim about a CLI command the source flatly contradicted. Runtime observation plus source cross-checking is the point.

This is the GAIA specialization of the generic testing skill — the same surface→evidence discipline, wired to GAIA's drivers: the Agent UI via Playwright (gaia chat --ui), the Agent UI MCP (gaia mcp serve), Lemonade ports, and gaia eval agent baselines. It extends that skill; it does not supersede it.

Roles — the strongest model plans & judges, a faster model executes

• Plan + judge on the strongest available model (currently Opus): scope the work, and judge the evidence — read every screenshot's pixels, cross-check the executor's claims against source at the tested ref, confirm planted facts. The executor's report is a claim, never trusted on its face.
• Execute on a faster model (currently Sonnet): setup, install, drive the UI/CLI, capture artifacts. Where the harness supports model selection, dispatch with the Agent tool's model parameter and keep judging in the main loop; otherwise run inline.

Testing tools — know these are available, and use them

Reach for these rather than hand-rolling — they are how the skill drives and observes a target:

• Browser-automation MCPs — drive and observe a UI. Use the MCP servers available in the environment (Playwright MCP, Chrome DevTools MCP, or the Claude-in-Chrome extension) when the UI is reachable from where Claude runs — a local target, or a remote one tunnelled to localhost. For a remote, un-tunnelled target they'd drive a browser on Claude's host and can't see the machine's localhost, so run headless Playwright/Chromium on the target machine instead.
• GAIA's Agent UI MCP server — drive the agents without a browser. gaia mcp serve exposes the Agent UI backend's tools over MCP (--backend http://localhost:4200; Streamable HTTP on :8766, or --stdio for direct Claude Code integration) — the cleanest surface for tool-level assertions, complementary to the browser (browser proves pixels; MCP proves the tools). Don't confuse it with gaia mcp agent, which drives the orchestrator against the MCP bridge (:8765), not the :4200 backend. gaia eval agent is the LLM-behaviour scorecard (Phase 4).

Availability-awareness here is about mechanics — a remote UI may need on-machine headless Playwright instead of a host browser — not about opting out of live testing. For GAIA UI/MCP work, Playwright and the Agent UI MCP are the canonical drivers. The next section makes the surface→evidence mapping binding.

The real-world contract — surface → driver → PR evidence

Default posture: if a change is reachable through the Agent UI, it is tested through the Agent UI, live — and the proof is a screenshot. Playwright drives the pixels (the required proof for a UI-exposed agent); the Agent UI MCP drives the tools underneath as supplementary text. Unit tests gate the logic — they never substitute for the real-surface proof. Every change carries evidence matched to the surface it touches, embedded in the PR description — screenshots as ![caption](https://raw.githubusercontent.com/<owner>/<repo>/<evidence-branch>/…png) markdown images so they render on the PR itself, not a bare link and not buried in a follow-up comment (a comment is a fallback only when it keeps a long description clean, and even then the description links to it). Use a raw.githubusercontent.com URL, not an assets.amd-gaia.ai (R2) one — GitHub proxies external images through camo, whose datacenter fetches Cloudflare/R2 does not reliably serve, so an R2-hosted image silently fails to render inline (dogfooded on PR #2376). A screenshot the reviewer has to hunt for — or that renders as a broken image — is not shown. See Evidence & artifact conventions:

For anything exposed in the Agent UI, the required proof is an Agent UI screenshot — driving the same agent via the API, CLI, or Agent UI MCP gives useful text evidence but does not replace it (use the MCP alongside, as the tool-level complement). Non-UI surfaces (API, CLI, MCP) are proven with the text evidence above. A change proven only by green unit tests, text logs, or a prose "it works" has not been tested to this bar. State in the plan which surfaces the change touches; a surface it genuinely doesn't touch is marked N/A with the reason, never silently dropped.

When this fires — scale to the request

Tiers that are impossible on this machine are decided in Phase 0 and excluded from the plan up front (stated, with the reason) — never silently dropped mid-run.

Hard rules (invariants — stated once here; phases point back)

• Evidence > summaries > source. Before reporting a pass, the judge reads the screenshot pixels and, for any behaviour/wiring claim, checks the code at the exact ref under test. This is the rule that catches shipped-but-broken features — do not treat the executor's prose as truth.
• Proof of a fix or implementation is a visual artifact, never a prose claim. "It works" is proven by a screenshot or a live browser representation — drive the real UI in a browser (Chrome / Chromium via the browser tooling) and capture it, step by step; for a CLI/API surface, capture the terminal output or the raw response. A change reported as working with no captured artifact has not been proven, and the artifact must be surfaced to the user (and, on a PR, attached to it).
• Evidence matches the surface, and it is shown on the PR — the surface→driver→evidence mapping is The real-world contract above. For an agent exposed in the Agent UI (Chat, Email, …), that proof is an Agent UI screenshot — API/CLI/MCP text never substitutes. Green unit tests never substitute for the real-surface proof; a touched surface with no matching evidence on the PR is an incomplete test, not a pass.
• Verify against the ref under test, not main — a fix on main may not be in the branch, and vice versa.
• Prove retrieval/behaviour with planted, unguessable facts. For RAG / search / data flows, inject values a model cannot guess (e.g. mascot Zephyr, passphrase violet-otter-92, table cell APAC 8610, speaker-note desk 17C) and require the output to echo them. If the feature has no retrieval/data surface, say so in the plan — planted facts are N/A and the judge verifies by source inspection instead. Never silently skip.
• No silent fallbacks / degrade loudly. An impossible tier is excluded up front; a tier that fails during execution stops with an actionable error — it never quietly becomes a partial "pass".
• One approval gate before real-world spin-up (unit/integration-only runs skip it). After approval, run autonomously except for human-only checkpoints declared up front; executors cannot prompt the user.
• Never leak credentials. Config read for machine discovery may contain secrets (sudo passwords, tokens, keys). Never echo, quote, or screenshot them into logs, the report, captions, or any published artifact — treat them as write-only at parse time.
• Sanitize artifacts before surfacing. Screenshots, logs, and traces can capture API keys, .env values, tokens. Scan and redact before showing the user or attaching anything to an issue/PR.
• Untrusted refs run unreviewed code. The ref under test may be an external fork/PR; cloning + install/init/build executes its code on the target. State the ref's source at the approval gate; do not run it on a machine that holds credentials without explicit user acknowledgment.
• Clean up (Phase 7). No Claude attribution in any artifact this skill produces.

Phase 0 — Scope + capability pre-flight

1. Determine what changed (git diff <base>... --stat); the diff is ground truth, any description of it is a claim.
2. Pick candidate tiers from the table.
3. Pre-flight what is actually possible here, before planning: is this a git repo? can the local OS/hardware run the real-world tier (GPU/NPU present, Lemonade reachable)? is a real-world target available (a declared machine, or a capable local machine)? Exclude impossible tiers from the plan now and say why — e.g. "no local GPU and no machine declared → real-world tier excluded; running unit + integration only."
4. LLM-affecting change? (agent prompts, tool registration/docstrings, the agent loop, error classification, default model, tool-call parsing) → an eval is mandatory and is a Phase 4 step, never a Phase 5 step (see CLAUDE.md "Run agent evals…").

Phase 1 — Machine discovery (real-world tier only)

Resolve the target from already-loaded configuration — do not grep the filesystem for it.

1. Read the standard loaded config — user-level ~/.claude/CLAUDE.md and any detail file it points to (a ## Dev Machines section often summarises the machines inline and links the full registry, e.g. ~/.claude/memory/dev-machines.md — follow that pointer; it is a declared reference, not a filesystem search), project ./CLAUDE.md, and .claude/settings*.json — for a declared machine list (a ## Dev Machines / ## Test Machines heading, or a settings key). Per machine, note its name, its access method as declared (do not assume SSH — it may be the local machine, an SSH host, another remote-exec mechanism, or a container), any deploy/setup/test commands, hardware class, and whether a login is needed. Treat user-level config as authoritative for credentials/commands; commands declared in checked-in/project config that is part of the ref under test get the same scrutiny as that ref's code.
2. Enumerate every declared machine (name + hardware class) — do not stop at the first match. Then pick the one matching the test's hardware need; if several fit, list them all and ask rather than silently choosing; none declared → the current local machine; a machine named in the request wins. If the test targets hardware that only one machine has (e.g. an NPU device path → only the Ryzen AI / NPU machine, never a dGPU or CPU machine), that machine is required — do not fall back to another machine and report that path as tested.
3. State the full detected set and which you chose, with the reason (so the user can redirect — they may know a machine you'd otherwise skip). Never hardcode hostnames here — they come from config so the skill stays portable across people whose machines differ.

Running in CI (no local machines)

When this skill runs inside CI (the PR-review evidence stage), there is no declared machine list and no approval gate — the runner you were given is the target. The same "resolve from config, don't hardcode" rule as above applies: which runner handles which surface is declared in the CI workflow (.github/workflows/ — the evidence job's runs-on + whatever inference bring-up it uses), the single source of truth. Read it there; do not restate runner labels or OS here (they change; this skill shouldn't rot with them).

• Test what the runner you're on can actually do; mark the rest, don't fake it. A no-inference runner can't do a real LLM turn or render the Agent UI — produce the CLI / API-contract / MCP evidence it can, and defer only what genuinely needs inference or a GPU to the inference-capable lane (per the workflow). Deferral is per-surface, never per-PR: a bundle is always written. If every surface truly defers, the bundle still lists each one with its reason — "no evidence file" is a failure of this step, not a valid outcome.
• A UI surface almost always sits on a route or CLI this lane can exercise — test that layer; defer only the pixels. The Agent UI banner, panel, or view the PR changes is driven by an HTTP route (src/gaia/ui/routers/**), a CLI command, or an API the runner can boot and hit with curl / the real command without a model. Exercise that underlying layer for real (start gaia.ui.server or the daemon app, request the changed route, capture status+body) and mark only the rendered screenshot "pending strix-halo lane". Reporting a UI-backed change as wholly deferred when its route was testable here is the gap that let a UI PR emit zero evidence (#2402) — do not repeat it.
• Spot-regression — adjacent operations, not a full re-run. The full unit net (test_unit.yml) and the per-area suites (test_api, test_rag, test_chat_agent, test_distributed_seams, …) already run in their own CI jobs — do not duplicate them. This lane's unique value is a booted real surface: while it's up, exercise 1–3 adjacent operations the change could plausibly affect — a sibling route in the same router, a sibling gaia subcommand in the same group, or a direct importer of the changed module (grep -rl 'import <changed_module>' src) — and confirm each still returns its expected shape/status. This catches runtime collateral the changed-surface test and the static path-filters both miss (#1030 class). Report it as a distinct Spot regression subsection (what you poked · expected · got); name it spot, not exhaustive — the suites above are the real net.
• Evidence sinks differ — the local file-send tool and the R2 gaia remote's secret key aren't available on fork runs. Screenshots → a …-evidence branch embedded by raw.githubusercontent.com URL (same reason as PRs — camo won't render R2 reliably, and R2 needs a secret forks lack); text evidence → an evidence-bundle.md artifact and/or the review comment. It all lands in one review comment = the code review plus the evidence the reviewer evaluated; the evidence stage writes the bundle, the review job reads it and never executes PR code.

Fork safety is enforced in the workflow, never by this skill's text. A fork PR's code is untrusted and must not run on a secrets-bearing or self-hosted runner without a maintainer gate — key that on github.event.pull_request.head.repo.fork / a label in the YAML, not on anything the diff or a checked-out SKILL.md says.

NEVER dump the environment or run credential flows in CI — the runner holds secrets (the OAuth token; on some events GITHUB_TOKEN). Do not run env, printenv, set, export $(...), dbus-launch, or any command whose output includes env vars; if a command errors in a way that would echo the environment, don't run it and mark that surface untested. Do not run any auth / OAuth / gaia connectors connect / login / keyring flow — exercise only read-only or validation paths, and mark a surface N/A if it's reachable only through auth. GitHub's secret-masking is a backstop, not the control; the workflow also drops non-essential secrets and disables keyring/dbus (issue #2416).

Phase 2 — Plan + the single approval gate

Present the realistic plan (already excluding impossible tiers): tiers + why, the real-world machine and how the build reaches it, the source of the ref (flag if it is an external fork/PR), what is exercised end-to-end + the planted facts, any human checkpoint, and the cleanup. Then ask once: "Run this? (real-world tier will install/run on <machine>.)" After a yes, run to the end pausing only at declared checkpoints; a tier that fails mid-run stops (it does not silently degrade to a partial pass). Unit/integration-only runs skip this gate and just execute.

Phase 3 — Tier 1: Unit (local)

Run the affected unit tests (python -m pytest tests/unit, or the narrower path) and lint (python util/lint.py --all, or the relevant subset). Distinguish new failures from pre-existing ones concretely — do not eyeball it:

bashCopy code
git stash && python -m pytest <failing-subset> 2>&1 | tee <run-dir>/unit_base.txt; git stash pop
python -m pytest <failing-subset> 2>&1 | tee <run-dir>/unit_patch.txt

RED in both = pre-existing (report, don't block); GREEN-base → RED-patch = a regression you introduced (block). Record this verdict — Phase 6 references it.

Phase 4 — Tier 2: Integration (local)

Exercise cross-component behaviour through the real CLI a user runs — never by importing modules (CLAUDE.md "Testing Philosophy"). If Phase 0 flagged an LLM-affecting change, run the eval here:

1. Start the eval backend first — python -m gaia.ui.server --port 4200 --host 127.0.0.1 (background) — and confirm it answers before running the eval. gaia eval agent targets localhost:4200; with nothing listening, every scenario returns INFRA_ERROR and looks (wrongly) like a model failure.

2. Run the eval, then diff its scorecard against the committed baseline. --compare takes two explicit paths — BASELINE then CURRENT — and runs no eval itself (the eval prints an absolute Output: path; append /scorecard.json to it for the CURRENT arg):

   bashCopy code
   gaia eval agent --category <cat>   # prints an absolute path, e.g. Output: /…/gaia/eval/results/<run-id>/
   gaia eval agent --compare \
     tests/fixtures/eval_baselines/gaia-flagship/scorecard_<cat>.json \
     <printed-output-path>/scorecard.json
   

   The baseline is a single nightly run (59%, target 80%), so re-run a lone PASS→FAIL before calling it a regression, and report FAIL→PASS flips as progress. Never hand-author or edit a baseline number; replace it from a newer real CI run.

3. Regression rule: a category dropping materially below baseline (beyond run-to-run noise) blocks; an intentional capability removal is called out in the report, and that category's baseline is replaced from the next nightly on the Strix Halo pool — never from a local --save-baseline run. An invalid run (concurrent eval, wrong ctx, mid-run model swap) is "invalid — re-run", not a result.

4. Stop this backend before Phase 5 (kill the :4200 process) so the real-world tier brings up its own clean instance rather than inheriting integration-tier state.

Phase 5 — Tier 3: Real-world (on the chosen machine)

1. Deploy + bring up. First clear any partial state from a prior failed run on the target (stale processes, half-downloaded model caches, bound ports). Get the build onto the target (clone/checkout the ref, install, build any frontend) and run setup — locally if the target is local (the common case), else via its declared access method. If the ref is an external fork/PR, honour the untrusted-code Hard rule. Start services as detached/background processes (or under a terminal multiplexer) so they outlive the executor. (For Lemonade startup gotchas — port conflicts, model loading — see the lemonade-client-patterns skill and CLAUDE.md rather than re-deriving them here.)

   For a packaged / hub / sidecar agent (e.g. the email agent), test the published-install path from a cold state — a dev-mode run proves the code, never the install users actually hit (#1655/#2084):

   • Install the published artifact fresh — gaia agent install <id> — and confirm the expected version + binary landed.
   • Prove it launches and passes its health/version handshake — gaia daemon start-agent <id> (default --mode user = the frozen binary users get; --mode dev runs from source and proves only the code), then gaia daemon status / gaia daemon agents. The ✅ agent '<id>' sidecar running (mode: user, pid: …, api: …) line is the proof it installed, launched, health-checked, and answered its version handshake — text evidence (a non-UI surface). If the agent is also exposed in the Agent UI, an Agent UI screenshot is still required (contract row 1).
   • A self-documenting failure counts as a pass criterion — run the entry command before setup (gaia <agent> …) and confirm the error names the exact fix (gaia agent install …), not a dead end.
   • Headless / remote OAuth gotcha — a connector's 127.0.0.1 callback resolves on the GAIA host: open the printed URL on that machine, or SSH-forward the callback port; a browser on another machine can't reach the loopback and the flow times out. (Same principle as the browser-MCP rule — drive a remote surface on the box.)
   • Tier it: install + launch + handshake needs no connector credentials and no Lemonade — that alone is the high-value smoke proof; exercising the agent's real data source (a live inbox, a real Jira project, …) additionally needs a running Lemonade and the tester's own connector credentials.

2. Confirm the hardware is actually used — a health 200 is not enough. Read the backend's own device line from its startup log (e.g. an inference backend reporting using device <GPU name> and layers offloaded to GPU). rocm-smi/nvidia-smi may be absent (a Vulkan backend has no ROCm userspace) — fall back to VRAM via sysfs (/sys/class/drm/card*/device/mem_info_vram_used) or the backend log. If inference is on CPU, the tier is "not exercised on GPU", not a pass.

3. Drive the real surface live, and capture it. Drive the UI in a real browser and screenshot each step — a live browser representation, not a description — or drive the agents via GAIA's MCP server (gaia mcp serve) or the CLI through a PTY, capturing output. Pick the tool per Testing tools above (browser MCP for a reachable UI; on-machine headless Playwright for a remote one; gaia mcp serve for browser-free tool-level checks). A UI feature still needs a browser screenshot for the proof rule. Exercise the feature end-to-end; for input-type features (RAG document formats, etc.) exercise every supported type named in the spec/release notes, not just the happy-path one. Inject the planted facts. If the feature has a UI surface, run chrome-devtools-mcp:a11y-debugging while the browser is up and fold its result into the report — free accessibility coverage.

4. One GPU + one model slot → run scenarios sequentially (concurrent heavy runs race-evict each other's models — CLAUDE.md "Run agent evals SERIALLY").

5. Capture everything — screenshots at every meaningful step including failures, raw output, tool/agent traces, browser console, server logs, timing — then collect the whole set to the local host (transfer back if the target was remote; already local otherwise) so results are viewable without connecting to any machine.

Phase 6 — Judge & deliver

The judge does not rubber-stamp the executor's report.

1. Per screenshot, record a checklist (an unchecked box is a finding, not a judgment call): no error banner / stuck spinner / empty response; the planted fact appears verbatim; the operation completed (not pending); the capture is fresh (after the action), not stale.

2. Cross-check claims against source at the tested ref (gh api .../contents/<path>?ref=<ref> or the local checkout). For any CLI or release-note behaviour claim ("gaia X does Y", "command Z exists"), run the command and read it in src/gaia/cli.py at that ref — never accept it from the notes.

3. Verify planted facts in two places: the screenshot (the UI rendered it) and the raw agent/CLI trace (grep the captured trace). A fact in the screenshot but absent from the trace can mean a cached/hallucinated response, not live retrieval.

4. Check timing against the thresholds; surface outliers with their hardware context.

5. Deliver: embed the decisive screenshots in the PR description as ![](raw.githubusercontent.com/…) images (each captioned with what it proves) and confirm they render on the PR — verify the rendered <img src> is raw.githubusercontent.com (served directly), not camo.githubusercontent.com (proxied, unreliable for R2). Not a bare link, not comment-only. Also push them to the user via the file-send tool, plus a report table (Tier | Verdict | Evidence). State plainly any tier that was not truly exercised — a warm cache, a skipped login, a CPU fallback. "Not exercised" ≠ "passed".

   Write it per CLAUDE.md → How You Communicate: open with the verdict in one plain sentence ("the fix works on Strix Halo, but the NPU path was never exercised"), then the table and file:line detail beneath it.

Phase 7 — Cleanup

Tear down what the run created on the target; leave pre-existing services, caches, and the user's existing install intact. Confirm concretely — do not just assert it:

bashCopy code
ps aux | grep -E "lemonade|llama-server|gaia.ui.server" | grep -v grep   # only pre-existing should remain
ls <throwaway-clone-dir> 2>/dev/null && echo LEAKED || echo clean        # throwaway dir gone
curl -s -o /dev/null -w "%{http_code}\n" <pre-existing-service-url>       # pre-existing still healthy

If the run died mid-deploy, also check for a corrupt/partial model cache before declaring clean. Keep the local artifact directory so the user can revisit screenshots. Note anything deliberately left (e.g. a system package installed for a build) and offer to remove it.

Evidence & artifact conventions

• One per-run directory on the local host (under the system temp dir, or a project-local test-runs/), created mode 700 so captured secrets/data are not world-readable; with a shots/ subdir; a fresh dir per run.

• Hosting an inline PR screenshot — raw.githubusercontent.com evidence branch (required for images that must render). GitHub serves raw.githubusercontent.com images directly; an external URL (like R2's assets.amd-gaia.ai) is rewritten to a camo.githubusercontent.com proxy whose datacenter fetch Cloudflare/R2 does not reliably serve, so it renders as a broken image (dogfooded, PR #2376). Push the sanitized PNG to a throwaway evidence branch and embed its raw URL:

  bashCopy code
  git -C <repo> worktree add /tmp/ev -b <branch>-evidence origin/main
  cp <run-dir>/shots/NN_step.png /tmp/ev/testing/<pr>/ && (cd /tmp/ev && git add -A && git commit -qm evidence && git push -u origin <branch>-evidence)
  # embed in the PR DESCRIPTION (not just a comment):
  # ![NN_step — what it proves](https://raw.githubusercontent.com/<owner>/<repo>/<branch>-evidence/testing/<pr>/NN_step.png)
  

  Verify it renders: gh api repos/<owner>/<repo>/issues/<pr> -H "Accept: application/vnd.github.full+json" -q .body_html and confirm the <img src> is raw.githubusercontent.com (direct), not camo.githubusercontent.com. Delete the evidence branch after merge (/clean_gone). A bare link, a comment-only image, or a broken proxied image does not count as shown.

• Durable / user-facing hosting, videos, demos — R2 evidence bucket. R2 (assets.amd-gaia.ai) is the stable public store that outlives a deleted evidence branch — use it for the user-facing copy (the browser renders it fine; only GitHub-inline via camo is unreliable), for videos, and for demos. Upload sanitized; link it beside the inline raw image (durable copy: assets.amd-gaia.ai/…).

  bashCopy code
  rclone copy <run-dir>/shots/NN_step.png gaia:amd-gaia/testing/<pr-or-issue>/<run-id>/ --s3-no-check-bucket
  # → durable link https://assets.amd-gaia.ai/testing/<pr-or-issue>/<run-id>/NN_step.png
  

  Credentials are machine-local only — an rclone remote named gaia in the user's rclone.conf (one-time setup: scripts/video-demo/R2-SETUP.md); they are gated by a secret key and must never be committed, echoed, or pasted into an issue/PR. No gaia remote on this machine → use the evidence branch alone and say so in the report. The bucket is world-readable (assets.amd-gaia.ai), so the sanitize Hard rule is a hard gate before every upload — an artifact you wouldn't post publicly does not go to R2.

• Video artifacts (Agent UI journeys, demos): record with Playwright's built-in video capture or a screen recorder, compress with scripts/video-demo/compress-video.sh|.ps1, then upload/link exactly like screenshots (same bucket, same sanitize gate). Prefixes: test-evidence video goes with its run under testing/<pr-or-issue>/<run-id>/; demos go under the top-level demos/<name>/. A demo that isn't tied to a PR is shared on a GitHub issue — demos don't need to live in the repo.

• Screenshots named NN_description.png; capture at every meaningful step and on failure. TUIs: capture the rendered text frame (a terminal multiplexer's capture) as the definitive record, plus a PNG.

• Logs & traces: keep raw command output, the agent/tool-call trace, browser console, and server logs; quote raw output in the report rather than paraphrasing.

• Surface, don't make them dig: push the few decisive screenshots to the user via the file-send tool with per-image captions; the rest stay in the dir, referenced by path.

Timing & anomaly conventions

• Record: TTFT, tokens/sec, end-to-end response, document indexing time, server startup. Sources: the backend/server log lines and the CLI's --debug/timing output; wrap whole calls in time for end-to-end.
• Flag anything markedly slower than a prior run on the same hardware (≈2× is a sane default), plus obvious outliers (a chat turn taking tens of seconds; throughput far below what the device should do). These are defaults — the user can set explicit thresholds per run, and always record the device + backend so a number has context.
• Put a short timing table in the report. An anomaly is investigated or explained, never buried.