report — did it work, and by how much?
The result of a run is not "we made edits" — it is a defensible answer to did this actually work, and by how much. report lays three numbers side by side: where the seed started (val), where the best candidate landed (val), and the single held-out test number that counts. It is what a human reads to decide whether to ship.
Runs standalone as /cap-evolve:report, or headlessly as the last step of
cap-evolve run — same scripts/run.py either way. report is a phase SCRIPT, not a
cap-evolve subcommand; invoke the script.
How to read the three numbers
The honest reading is always test vs baseline, with val as a sanity check in
between. scripts/run.py produces the numbers; this is the judgment you add on top:
- test ≈ baseline → no real gain. The val improvement was overfitting or noise
the gate let through. Do not ship; tighten
gate_k_seor add trials. - test ≫ baseline → genuine improvement on data the optimizer never saw. Ship.
- val ≫ test → the classic overfit signature: the optimizer learned the val set, not the capability. The reported val→test gap is the overfitting, quantified.
- pass^k far below pass^1 → the gain is fragile across trials; the agent sometimes succeeds but not reliably. A high mean with low pass^k is not a dependable win (τ-bench's point).
Every number is rendered with its stderr when one was measured, because "0.71" and "0.71 ± 0.08" support very different decisions. A gain smaller than the noise floor is not a result — say so plainly rather than quoting the point estimate alone.
Output contract
scripts/run.py owns both artifacts and writes them deterministically from the run
dir — do not hand-write or paraphrase them, or two runs stop being comparable.
report.md is exactly this skeleton (bracketed lines appear only when they apply):
# cap-evolve run report — <run_id> [> **NOT FINALIZED** — no held-out test number. Run the finalize phase first; …] [> **No holdout** (train == val == test). The test number below is a *fit* metric, …] - Best candidate: `<best_id>` - Baseline val: <r> ± <se> - Best val: <r> ± <se> - **Held-out test (optimized skills): <r> ± <se>** (pass^1=…, pass^k=…) [- Held-out test (baseline `<baseline_id>` skills): <r> ± <se>] [- **Test improvement (optimized − baseline): <+Δ>**] [- Val→test gap: <+Δ> — selection optimism on val; this gap IS the overfitting] - Iterations: <n> [- Optimized for: <consuming model> (tier <t>)] [<sealed note — omitted entirely when the run was never finalized>]
stdout is exactly one JSON object — cap-evolve run echoes it as its own result, so
it is the machine contract for everything downstream. Keys (null for whatever the run
dir does not carry; an unfinalized run is finalized: false with null test numbers):
run_dir, best_id, finalized bool, no_holdout bool, baseline_val,
baseline_val_stderr, best_val, test_reward, test_stderr,
test_baseline_reward, test_baseline_stderr, test_delta, test_pass_k (k→float),
val_test_gap, iterations, target_profile ({model,tier,resolution_note}), plus
dashboard / dashboard_server / *_error on the paths that produce them.
How to run
python scripts/run.py --run-dir .capevolve/run_XXXX # JSON + report.md + dashboard.html python scripts/run.py --run-dir .capevolve/run_XXXX --terminal # colored in-chat ANSI report python scripts/run.py --run-dir .capevolve/run_XXXX --no-dashboard
--dashboard-mode / --dashboard-port / --dashboard-url are orchestrator-supplied.
Re-reporting by hand after cap-evolve run needs --dashboard-url <the URL run printed>
or --no-dashboard — launching is deliberately not idempotent, so a bare re-run spawns
a second server on a second port and reports that one instead.
The dashboard (dashboard.html)
One self-contained static file (inline CSS/JS/SVG, no CDN, no server, no network — opens
from file://); the single shareable artifact. Eight panels reduced from the event log
plus baseline/final, rollouts and the git store; every value passes a recursive secret
redactor so a shared dashboard leaks no API keys; optional panels degrade silently when
per-task data, diffs, or finalize are missing. --terminal renders the same reduction
as an ANSI chart for in-chat progress.
References
references/concepts.md— why the val→test gap measures overfitting, pass^k fragility, reporting uncertainty, with sources. Load when writing the human interpretation and you want the reasoning or a citation.references/dashboard.md— the reduced graph + summary schema, per-panel field sources,--terminal, redaction, degradation matrix. Load when changing or debugging the dashboard; not needed to run the phase.
