Prompt Regression Suite Skill
Every prompt tweak, model upgrade, and context change is a deploy. This skill designs the suite that runs on each one and answers a single question: did anything that used to work stop working?
What This Skill Produces
- A golden case set: curated inputs with per-case pass criteria
- Scoring methods per case class (exact, rubric-judge, property checks)
- CI gate thresholds — what blocks a merge vs. what warns
- A failure triage protocol — flaky vs. regressed vs. golden-set-wrong
Required Inputs
Ask for (if not already provided):
- The feature and its contract — what the LLM step receives and must produce
- What has broken before (or nearly) — past incidents seed the best cases
- Real traffic examples — 10-20 representative inputs, including ugly ones
- What triggers a run — prompt edits, model bumps, retrieval changes, all of the above?
Building the Golden Set
Compose the set from four deliberate classes — not a random sample:
| Class | Purpose | Share |
|---|---|---|
| Core paths | The 5-10 inputs that represent most real traffic | ~40% |
| Past failures | Every input that caused a bug, complaint, or incident — permanently | ~25% |
| Edge & adversarial | Empty/huge inputs, wrong language, injection attempts, off-topic | ~25% |
| Canaries | Cases pinned to behaviours you never want to change (refusals, format, tone) | ~10% |
Keep it small enough to run on every change (30-80 cases beats 500 nobody runs). Version it in git next to the prompt.
Scoring Per Case
Choose the cheapest check that catches the regression:
- Exact / structural — JSON parses, required fields present, enum values legal. Free and deterministic; use wherever the contract is structural.
- Property checks — output contains/never-contains X, length bounds, citation count. Deterministic proxies for quality.
- LLM-as-judge with a rubric — only where judgement is unavoidable. Pin the judge model + rubric version, score against the baseline output, and spot-check judge agreement with a human on ~20 cases before trusting it.
Every case records: input, pass criteria, scoring method, and the baseline output at the time it was added.
CI Gates
- Block the merge: any past-failure or canary case fails; structural pass rate < 100%; overall pass rate drops more than [X]% vs. baseline.
- Warn, don't block: judge-scored quality drifts within tolerance; latency/cost moves past its soft budget (pair with
llm-cost-latency-budget). - Every run logs model ID, prompt version, and per-case results — regressions must be diffable to the exact change.
Failure Triage Protocol
When a case fails, classify before "fixing":
- Flaky — re-run N times; if intermittent, tighten the prompt/temperature or the check, don't ignore it.
- Genuine regression — the change made it worse: revert or fix the change.
- Golden set wrong — the new behaviour is actually better: update the case via review, never silently, and record why the expectation changed.
Output Format
Prompt Regression Suite: [feature]
Trigger: runs on [prompt edit / model bump / retrieval change] via [CI job].
Golden set ([n] cases):
| # | Class | Input (summary) | Pass criteria | Method |
|---|
Gates: merge blocks when [conditions]. Warnings on [conditions].
Triage: [the three-way protocol, with who owns updates to the golden set]
Maintenance: every production incident adds a case within [period]; the set is reviewed for dead cases each [quarter].
Quality Checks
- Every past production failure appears as a permanent case
- Canary cases cover the behaviours that must never change (refusals, format, safety)
- No case relies on an LLM judge where a structural or property check would do
- Gate thresholds are numbers, not "significant degradation"
- The suite is fast and cheap enough that it actually runs on every change — state its runtime and cost
Anti-Patterns
- Do not test only happy paths — the suite exists for the inputs that hurt you
- Do not let anyone update golden expectations in the same PR that broke them, without review
- Do not use an unpinned judge model — a judge that upgrades itself moves your baseline silently
- Do not treat pass-rate-vs-baseline as the only gate — one dead canary matters more than 2% aggregate drift
- Do not grow the set unboundedly — a suite too slow to run on every change protects nothing
Example Trigger Phrases
- "Stop prompt changes breaking production."
- "Set up golden tests or CI gates for an LLM feature."
- "Test a model/prompt upgrade before shipping it."
