Benchmarking the flagship agent
gaia eval tasks runs the same tasks through GAIA and through Claude Code,
scores them mechanically, then has an LLM judge grade the transcripts. It
answers three questions: does the work get done, is the answer honest
about it, and what did it cost.
Everything here was learned by running it. The traps section is the part that
saves a day. The full reference is
docs/reference/eval.mdx.
The suites
| Suite | What it is |
|---|---|
everyday | The 14 tasks a developer actually asks for. The comparison table's suite. |
therock | Four real merged fixes in TheRock, from the commit before the fix. Two forbid the internet in the prompt. |
core / full / adversarial | What CI gates on: the traps and the committed expectations. |
The one rule: serial
Every run wants the local Lemonade model slot, and two at once race-evict each
other's models — the same reason CLAUDE.md gives for gaia eval agent. Before
starting anything:
bashpgrep -fl "[g]aia eval tasks" | wc -l # must be 0
Budget 4–9 minutes per model for the 14-task suite depending on the model's
speed, plus about 2 minutes to judge it. Never pkill a run mid-task, and
never stop Lemonade or the daemon while one is in flight.
Recipes
One model, three runs, real cost. The main loop of any harness change:
bashgaia eval tasks run --suite everyday --model fireworks.glm-5p3-flash \ --repeats 3 --meter fireworks --out runs/gaia-glm
The same model through Claude Code, so the difference is the harness:
bashgaia eval tasks run --suite everyday --harness claude-code \ --model fireworks.glm-5p3-flash --repeats 3 --out runs/cc-glm
Claude Code on Anthropic's own model, as the reference row. Run Opus 5, not Sonnet: it is the strongest leg available, so a cost share reads as a fraction of the best agent and it sets an honest quality ceiling.
bashgaia eval tasks run --suite everyday --harness claude-code --model claude-opus-5 \ --out runs/cc-opus
The TheRock tier, a real 144K-line codebase:
bashgaia eval tasks run --suite therock --model fireworks.glm-5p3-flash --out runs/gaia-therock
The table. The first directory is the 100% row:
bashgaia eval tasks report runs/cc-opus runs/gaia-glm runs/cc-glm --out runs/report
The judge's own controls, before you trust any quality number:
bashgaia eval tasks controls
Reading a result honestly
- One run is noise. A single model's quality spans about 0.3 across repeat
runs. Never report a delta below that from n=1 — use
--repeats 3, and read the min–max the table prints under each mean. A gap smaller than the range it sits in is noise. - Cost is three different kinds of dollar.
meteredis real money;api_equivalentis Claude Code's list-price figure on a subscription, which is a price of compute and not money spent;harness_countsis a rate-carded estimate. The reports label them. Never sum or rank them together without saying which is which. - Per-task deltas locate a regression; the aggregate only tells you one
exists. Diff the per-task rows between two runs and read the judge's
one_linefor the worst. - Then check the transcript, not your theory.
gaia.eval.bench.transcripts.tool_calls(transcript)yields(name, args, result)for either harness. Count what the agent actually did before blaming a change: twice a confident hypothesis (trimmed tool descriptions; tool ordering) died on inspection. - Watch for confounds. After compound shell commands landed,
ghinvocations halved while the same work got done in chained calls. A raw call-count drop is not a behaviour regression.
Traps that cost real time
A wall-clock cap silently penalises slow models. At a 420 s cap, GLM-5.3
full averaged 182 s per task, got cut off three times, scored 40% and looked
incapable. The default is 1800 s; keep --run-timeout the same for every model
in a comparison and high enough that none of them binds.
Rate cards drift, and a wrong one rewrites every cost claim. A table once
had DeepSeek V4.1 Flash at $0.22/$0.007/$0.66 when the published card was
$0.30/$0.006/$1.20 — every Flash cost was ~30% low. Verify against
docs.fireworks.ai/serverless/pricing before quoting cost. Cards live in
src/gaia/eval/bench/metering.py::RATES; a model with no published card prints
n/a, never a guess. --meter fireworks bypasses the question entirely.
A metered run is account-wide. Two metered runs on one account at the same time pollute each other's deltas, and the meter trails by ~90 s.
Keyword rubrics are both lenient and brittle. They pass an answer
containing the right word without it establishing the point, and fail a correct
answer phrased differently. must_establish judged by the LLM replaced them.
Do not add new keyword checks.
The judge often costs more than the run it grades. A 14-task judge pass is
$0.4–0.9; GLM-5.3 Flash's entire run is $0.083. Do not re-judge a run you have
already judged — judge is a separate command for that reason.
Fresh input dominates GAIA's cost, and the first call of every task is uncached because each task is a new process. Within a process caching reaches ~95%; across processes Fireworks gives a new one nothing, while Claude Code's account-wide cache hits 91% across its runs. That asymmetry flatters Claude Code's token efficiency and is worth stating when reporting.
Graders must not write into the agent's workdir. A probe that writes
data.txt makes the judge blame the agent for a stray file. Grading runs after
the agent exits, and the diff is taken before the probes run — keep it that way
when adding a probe.
Adding a model
- Confirm the gateway can reach it:
gaia eval tasks run --suite everyday --tasks 21-qa --model <id> --no-judge. - Add its published card to
metering.py::RATES, keyed by the bare model name, or meter it with--meter fireworks. - Run it alongside at least one model you already have numbers for, so a harness change cannot be mistaken for a model difference.
What good looks like
Numbers below are a snapshot from September 2026 — treat them as the shape to expect, and re-measure rather than quoting them.
On the 14-task suite, Claude Code at Opus 5 scored 14/14 at quality 4.97, truthfulness 5.00, 83 steps, 456 s. Against that ceiling GLM-5.3 Flash on GAIA reached 14/14 at quality 4.96 for $0.09 of real Fireworks charges, and DeepSeek V4.1 Flash 4.88 for $0.105. A model that costs more than the Sonnet reference and scores below it is not a candidate, however good its public benchmarks look.
Truthfulness is where the daylight is: the best legs score a clean 5.00, most others sit at 4.86. That axis — never claiming what the tool record does not support — is the one worth optimising next.
