hill-climb — one loop, three focus schedules
Greedy search over candidates: the parent is always the run's current best, and a
child replaces it only by clearing the val significance gate. The test split is
never touched here — that is finalize.
Requires baseline first. Without --resume the loop reads the seed's val
result from <run-dir>/baseline.json (scripts/run.py:121-122); with no run state
it raises FileNotFoundError: no run state at .../state.json. --resume instead
reads the current best's val from its stored rollouts, falling back to
baseline.json when the run has no best yet (run.py:118-122).
One iteration, end to end
This is the mechanism the other algorithm skills vary; they describe only their
differences and point back here. One iteration is harness.run_step
(core/cap_evolve/harness.py:1409):
- Pick the parent — always the current best (
harness.py:2224,run_dir.candidate_dir(run_dir.best_id)). Copy it towork/<cand_id>/; the optimizer edits that copy in place, so the parent is never mutated (harness.py:1454-1458). - Build the prompt. The parent's val per-task rows are split into
always-failing / flaky / infra-errored / solid (
harness.py:1775-1795), rendered as the failure index plus an explicit protect these passing ids block (harness.py:1803-1879), and substituted into the project's optimizer-instructions template.--focusnarrows which failures are emphasized; nothing else changes. - Inject context and memory. Full trajectories, capability guidance, and the four
cross-iteration files land in the workdir (
harness.py:1062-1094) — seereferences/run-step.md. - Optimize. The optimizer command mutates the workdir. A crash is caught, logged,
and left as an unchanged copy of the parent, so the gate simply rejects it — a
wasted iteration, not a dead run (
harness.py:1486-1500). - Evaluate on val only (
harness.py:1516), at--n-trialstrials per task. - Gate. With per-task data on both sides the paired test is chosen automatically:
accept iff mean per-task Δ >
k·SE of those paired deltas (harness.py:1524-1532).--no-regressionadds a second, harder condition on top. - Commit. Every candidate is snapshotted — accepted and rejected — so any
iteration can be diffed (
harness.py:1557); the version store commits it (harness.py:1609-1612). Only an accepted candidate callsset_bestand becomes the next parent (harness.py:1558-1559); a rejected one is filed in the rejected memory that feeds the next prompt (harness.py:1607-1608).
Why the parent is always the current best. The gate already guarantees every
accepted candidate is a real improvement on val, so the best candidate is the only
one with evidence behind it — forking anything else spends budget on a lineage
already known to be worse. The cost is that a candidate which trades one task class
for another can never be kept as a specialist; wanting that is the reason to use
gepa, whose per-instance Pareto frontier keeps specialists on purpose.
Why the bar is Δ > k·SE, not Δ > 0. Rewards are estimates from a finite sample
of tasks and trials, so about half of all no-op edits measure as a small positive
Δ by chance. Accepting on Δ > 0 therefore ratchets on noise: val creeps up, the
sealed test does not move, and the run reports a gain that was never there. The bar
is the noise scale itself, so a win has to be larger than the measurement error that
produced it. phases/gate owns the full statement of the decision and its modes.
Focus schedules
--focus | what each iteration emphasizes | when to use |
|---|---|---|
all (default) | every failing val task — find the one edit that lifts the most | broad capability gaps; the usual choice |
cyclic | one val task at a time, round-robin | many distinct, unrelated failure modes |
hardest-first | val tasks ordered by the parent's per-task reward ascending, then cycling | a few very hard tasks dominate the gap |
All three index the val per-task results, because those are the only per-task
data the loop holds. Non-regression protection covers the whole val split under
every schedule, not just the focused task. hardest-first costs no extra evaluation:
it orders off the per-task rewards already in hand.
Back-compat: --focus all-at-once is accepted and treated as all (run.py:35).
Key flags
Beyond --run-dir / --project / --optimizer / --focus / --max-iterations /
--n-trials / --resume (scripts/run.py):
--gate-mode(defaultauto) +--k-se(default0.2) —autolets the engine pick the paired gate;significant|paired|strict|thresholdpin it. Raisingk-semakes acceptance stricter.--protected-paths— globs sealing the eval surface (scorer, gold, tasks, splits;defaultexpands to the built-in set). A candidate that edits one is indecisive, never scored: the measurement would grade a compromised harness, so no reward is recorded, the stall counter is untouched, and best is unchanged (harness.py:1440-1446). Leave this on for any run whose number you intend to quote.--capabilities— comma-separated capability skills under optimization. When empty the optimizer receives no allowed-edit-space block at all (harness.py:1735-1737), so it guesses the edit surface from the files.--no-regression— reject a candidate that lowers any val task the parent scored higher on, even when the mean improves.--convergence— graded plateau signal (ok→warn→paradigm_shift→stop) appended to the prompt, so a plateau escalates the ask instead of burning the remaining iterations on more of what failed.--workers N— concurrent rollouts per evaluation. Only safe when the adapter'srun_targetis thread-safe; a shared client, temp path, or cwd will corrupt scores rather than fail loudly.--store git|copy|command(+--store-commit-cmd) — how each iteration is versioned; git is the default and every candidate becomes a commit.--instructions-file,--bench-repo,--capability-sources,--optimizer-name,--target-model,--target-profile-file— prompt and read-context wiring;cap-evolve runfills these from the spec.
Standalone use
bashpython scripts/run.py --run-dir .capevolve/run_X --project .capevolve/project \ --optimizer 'python .../run-optimizer/scripts/run.py --name mock --workdir {workdir} --prompt {prompt}' \ --focus hardest-first --max-iterations 10 --n-trials 4 --protected-paths default
Known gate edge case (open, issue #351)
Under the paired gate with k_se = 1.0, a candidate that improves exactly one
val task and changes nothing else has Δ̄ == SE(Δ) algebraically, so a strict >
resolves it on floating-point representation alone. The default k_se = 0.2 banks such a
one-task gain (the bar is 0.2·Δ̄); at k_se = 1.0 expect it not to bank, unpredictably
by split size. Prefer edits that generalize across a class of tasks, which is what the
loop is asking for anyway.
Agent mode
When orchestration_mode: agent, drive the loop yourself with the same mechanism as
above: parent = current best, one edit per iteration, evaluate on val, gate,
accept → snapshot / reject → revert, seal once with the finalize phase script
(skills/phases/finalize/scripts/run.py), then report. orchestrate owns the
agent-mode rules; the hill-climb-specific obligation
is that you must reproduce the handover surface run_step normally builds —
LEDGER.md, JOURNAL.md, PROCESS.md, RUNMAP.md + prior_iterations/ — and carry
rejected edits into the next iteration's prompt. Skip it and the dashboard goes dark
and the optimizer re-proposes edits already refuted.
Helpers in the surface (v2, off by default)
When helpers.enabled and the surface allows executable files, the proposer should codify a recurring procedure
into a deterministic helper once it fails or recurs in 2 or more distinct feedback traces, instead of adding more
prose. A helper must be a general, offline, deterministic function; the pregate lint and helper_smoke reject
task ids, copied answers, network use and nondeterminism before any rollout is charged. Details:
references/helpers.md.
References
references/helpers.md— when and how to write a helper, and the checks it must pass. Load when the surface allows scripts and a procedure recurs.references/run-step.md— the shared step's exact contract: the handover files and their ownership, the rejected/accepted memory, the version store, the snapshot filter, and the tamper path. Load it when you need the contract verbatim, or when writing an algorithm that reusesrun_step. The sibling algorithm skills link this file rather than this body.references/focus-schedules.md— how each schedule builds its focus set. Load when choosing between the three or debugging a focus set.
