Running Benchmarks
Overview
You are the machine the producer ran so a perf regression wouldn't reach a human — or production — first. Benchmarks are backpressure: they say "no" to a slow patch at the boundary where it was introduced, not three releases later when someone files a latency bug.
Core principle: a benchmark number is meaningless on its own. A single "after" run proves nothing. A verdict requires a baseline to compare against and a way to separate a real change from run-to-run noise. Everything below exists to enforce those two things, because the natural temptation is to run once and eyeball it — and that is worthless.
When to Use
- A
backpressuredloop is on a performance-sensitive project and needs to confirm a patch didn't regress perf — a quick targeted check each iteration, and the full suite before leaving the loop.
Not for: projects with no benchmark suite or no perf sensitivity — skip benchmarking entirely rather than inventing it. This is not a correctness check; tests cover that.
Step 1 — Discover the suites and their time budgets
If the project has a BACKPRESSURE.md, check it first — it may name the suite(s) and exact commands to use. Otherwise find the commands yourself — package.json scripts, Makefile, a benchmarks//bench/ dir, CI config, README. Identify which suites exist and roughly how long each takes. Expect a spread: fast micro/targeted suites (seconds) and full end-to-end suites (minutes). Note what each one actually measures so you can map a suite to the code path you touched.
Step 2 — Pick the suite for the moment
- Each iteration: run the fast, targeted suite that covers the path you changed. It must be cheap enough to run after every patch without rabbit-holing.
- Before done: run the full suite once. Targeted-green is necessary, not sufficient — a change can help one path and regress another.
Don't run an 8-minute end-to-end suite to check a change that only touches one micro path; don't declare done off a micro suite alone.
Step 3 — Always measure baseline vs patched
Never judge from the patched numbers alone. Run the unchanged baseline (git stash, or check out the parent commit / a worktree) and your change back-to-back, on the same machine under the same conditions. The baseline is the only thing that makes the "after" number mean anything.
Quiet the machine first — close background load, don't run benchmarks mid-build or while other heavy processes compete. Uncontrolled machine state is the largest real source of noise, and "same conditions" means actually the same, not just same hardware.
Step 4 — Beat the noise with repetition
Benchmark output is noisy (often ±3–5% run to run). One run per side is not evidence.
- Run each side multiple times (≥5 when the suite is cheap).
- Compare the median (and the min for CPU-bound microbenchmarks — least contended, most stable), not a single sample.
- Establish the noise band from the spread of the repeated runs (or a known per-suite figure). The band is your threshold for everything below.
Step 5 — The verdict: regression / improvement / wash
| Verdict | Condition | Action |
|---|---|---|
| Wash (neutral) | Patched median within the noise band of baseline; distributions overlap | Passing. Move on. |
| Improvement | Patched consistently faster, gap clears the noise band | Passing. Note the win. |
| Regression | Patched consistently slower, gap clears the noise band, distributions don't overlap | FAIL. Fix it, or stop and name the blocker — never wave it through. |
| Ambiguous | Result straddles the band (e.g. ~noise ± a little) | Not a verdict yet. Add runs (≥10/side) and/or run the full suite to resolve before deciding. |
If a suspicious result still won't separate from noise after more runs — a small true shift that the suite genuinely can't resolve — do not default it to a pass. Treat it as a blocker and name it (the change may need a less noisy benchmark, or a human call), exactly as you would any other unresolved check.
Step 6 — Record the evidence
"No regression" asserted with no numbers is not a passing check. Record baseline vs patched, run count, and the median/min you compared. Evidence before assertion — the same standard the rest of the loop holds you to.
Common rationalizations
| Rationalization | Reality |
|---|---|
| "I ran it once, the number looks fine" | One run inside a ±5% noise band tells you nothing. Baseline + repeats or it doesn't count. |
| "It's a tiny bit slower, probably noise" | "Probably noise" is a measurement you didn't take. If it straddles the band, add runs until it resolves. |
| "The micro suite is green, we're done" | Micro-green can hide a full-suite regression. Run the full suite before done. |
| "Running the baseline too is double the work" | Without a baseline the patched number is uninterpretable. Half the work producing nothing is not a saving. |
| "I'll just trust the full suite once at the end" | A regression found at the end is a bisect. Run the targeted suite each iteration. |
Red flags — STOP
- Reporting a benchmark result with no baseline to compare against.
- Declaring "no regression" with no recorded numbers.
- Calling an ambiguous straddle-the-band result a pass instead of adding runs.
- Skipping the full suite before done because a targeted suite was green.
- Treating a real, beyond-noise slowdown as acceptable without fixing it or naming the blocker.
