Gepa

gepa — sample-efficient reflective Pareto search

algorithms/hill-climb owns the mechanics every algorithm shares: parent → proposal → val gate → commit, specified once in algorithms/hill-climb/references/run-step.md. Read that first. This page states only what GEPA (Agrawal et al., 2025) does differently, and why those differences are the paper's actual contribution rather than decoration. A thin wrapper over cap_evolve.gepa.gepa_loop.

The two mechanisms, and why removing either turns GEPA back into hill-climb

1. The parent is sampled from per-instance winners, not from the global best. A mean is a lossy summary. A candidate that fixes one genuinely hard task while regressing three easy ones has a worse mean than the incumbent, so a best-parent rule discards it — and with it the only text in the pool that has ever solved that task. GEPA instead scores per val instance and samples frequency-weighted over candidates that (co-)win at least one, so specialists and stepping-stones stay reachable as parents while their mean is still behind. That is the quality-diversity argument (MAP-Elites): keep the set that covers the task distribution, not the single champion. Sampling is stochastic and seeded, so the exploration is reproducible.

2. A cheap train minibatch pre-gates the expensive val evaluation. Rollouts dominate cost and a full-val eval costs |val| · n_trials of them. Most proposals are bad; paying full price to find that out is what makes naive reflective search unaffordable, and GEPA's headline "~35× fewer rollouts" comes almost entirely from not paying it. So parent and child are evaluated on the same small train minibatch (2 · minibatch-size rollouts, eval-cached) and the child is dropped unless sum(child) > sum(parent). The minibatch never decides acceptance — it decides whether acceptance is worth measuring.

A side benefit of (2): reflection reads train traces, so the proposer never sees the split its gate is computed on.

What differs from hill-climb, step by step

1. Parent — frequency-weighted sample over per-instance (co-)winners (--selection-strategy, default pareto_per_instance), not the current best.
2. Signal — a minibatch of --minibatch-size (default 4) train ids, evaluated with traces, instead of the whole train focus set.
3. Reflective dataset — REFLECTION.md in the optimizer workdir, over the parent's failing minibatch tasks (phases/diagnose owns what one is and what shape it takes). "Failing" is the hard threshold reward < 1.0, so with a graded scorer that never reaches 1.0 every sampled task is listed and the header always reads 0/N pass — read it as "sampled tasks, worst first". Each entry is truncated to ~800 chars and at most 12 tasks are written: a summary, not an archive; untruncated rollouts stay in rollouts/train/. The prompt also carries the run's cross-iteration files (LEDGER.md, JOURNAL.md, PROCESS.md, RUNMAP.md + prior_iterations/) so a proposal builds on prior work. All five are real here since #396: harness.record_iteration writes the step event those files are built from and folds the optimizer's appended JOURNAL.md entry back into the run-level handover, so the history accumulates across iterations.
4. Local gate — child on the same minibatch, sum(child) > sum(parent), else dropped with no val spend. This is the extra stage; everything after it is hill-climb's.
5. Merge — every --merge-cadence accepts, find two strict-frontier dominators sharing a common ancestor both beat and recombine them component-by-component (each component from whichever descendant changed it), then minibatch-gate and val-gate the result like any other child.

Note the word "frontier" covers two different sets here: the sampling pool in step 1 is every candidate with ≥1 instance win, which can include dominated candidates; the strict per-task Pareto frontier (selection.pareto_frontier) is a subset of it and is what the merge and the reported frontier_size use.

Component selection

A component is one editable file of the candidate. (Unrelated to hill-climb's --focus, which selects tasks; this selects files.)

• --component-selector round_robin (default): one component per iteration, cycled, written to FOCUS.md. Small attributable changes are exactly the unit the merge can later recombine — a sprawling multi-file rewrite cannot be.
• --component-selector all: list every component; the optimizer may edit anywhere. Use for monolithic capabilities or genuinely cross-cutting changes.

For a single-file capability the two coincide and the merge skips gracefully (gepa_merge_skip) rather than emitting a degenerate child.

Key hyperparameters

• --max-metric-calls (default 0 = unlimited): PRIMARY budget, checked between iterations. An in-flight iteration runs to completion, so actual spend can exceed it by up to 2·minibatch-size + |val|·n-trials — and a merge fires inside an accepting iteration, adding 3·minibatch-size (the merge and BOTH parents, on a freshly sampled minibatch) plus a second |val|·n-trials, for a worst case of 5·minibatch-size + 2·|val|·n-trials. Set it below your hard ceiling.
• --max-iterations (default 50): secondary cap on propose→gate iterations.
• --minibatch-size (default 4): train ids per cheap local gate.
• --n-trials (default 1): rollouts/task on the full-val eval (raise under noise so the significance gate is trustworthy). Minibatch evals are always 1 trial.
• --max-merges (default 2): cap on merge attempts that built a candidate — a merge rejected at either gate consumes one. A skip (no eligible pair) is free.
• --merge-cadence (default 3): accepts between merge attempts.
• --protected-paths (empty = off; default = the built-in globs): seals the eval surface (scorer/gold/tasks/tests). A child that edits one is INDECISIVE — no reward recorded, not remembered as rejected, stall counter untouched — because scoring a gold-hacking edit at all would teach the optimizer that it worked. It still charges spent.iterations (record_iteration(..., indecisive=True)): the rollouts and the optimizer call were really spent, so the spend meter counts it and only the evidence meter does not.
• --workers (default 1): pools the minibatch rollouts. Only safe when the adapter's run_target is thread-safe.
• --store / --store-commit-cmd (default git): where accepted candidates are committed.
• --gate-mode / --k-se, --no-regression, --seed: as hill-climb.
• --resume: rebuild pool/lineage/frontier from gepa_state.json + each accepted candidate's rollouts and continue the search. Preserved spend keeps the budget honest; the parent-sampling RNG stream restarts, so a resumed run is not byte-identical.

Known gaps (present tense — the shipped loop, not the paper)

Two. Re-derived against current main, because most of what this section used to list has since been fixed in core: the hollow eval-cache reflection by #387, the missing step record and the non-accumulating JOURNAL.md by #396 (harness.record_iteration is now the one place every algorithm ends an iteration, and GEPA calls it), and the dirty snapshots plus the un-excluded optimizer-agent dotfiles by #350 and #386. Check the two below before trusting them.

• The reflective dataset does not carry the task input, though _write_reflection's own docstring says each failing task "contributes its input" (gepa.py:230). It writes Agent output / Trajectory / Feedback and no input field (:263-267), so the optimizer sees a bad answer to a question it cannot read. Since #387 the output and trace survive an eval-cache hit, so the entry is no longer hollow — just anonymous.
• If splits.train is empty the minibatch silently falls back to val ids (gepa.py:530), putting the gate split in front of the proposer, with no warning. Do not run GEPA with a zero-size train split.

How to run

bashCopy code
python scripts/check.py    # behavioral, offline (mock optimizer + synthetic adapter)
python scripts/run.py --run-dir .capevolve/run_X --project .capevolve/project \
  --optimizer 'python .../run-optimizer/scripts/run.py --name mock --workdir {workdir} --prompt {prompt}' \
  --max-metric-calls 400 --minibatch-size 4 --component-selector round_robin

Requires baseline first (reads the seed's full-val result from baseline.json). Reports the pool, frontier_size, best candidate, accepts, merges, and metric-calls spent.

Agent-mode loop

When orchestration_mode: agent, follow orchestrate/orchestrate §Agent-mode loop for the shared rules, and make each round GEPA-shaped: pick the parent by per-instance win count; sample minibatch-size train ids; evaluate parent then child on that same minibatch and drop the child unless sum(child) > sum(parent); only then pay for a full val eval and its gate. Reflect on the train minibatch, never on val — val is the judge, not the teacher.

References

• references/concepts.md — the paper's thesis (language as a richer learning medium than a scalar), the frequency-weighted per-instance frontier, the system-aware merge and its tie-breaking, the metric-call/eval-cache accounting, and how the pieces relate to the hill-climb / skillopt siblings. Load when you need the reasoning behind a knob rather than its value. Cites arXiv:2507.19457.