Verify Cascade

Verify-Cascade — extract cheap, verify, escalate only on a flag

maggy already routes by task complexity. This makes routing output-aware: a cheap model does the work, a decomposed verifier checks it, and you escalate to a strong model (Claude) only when the verifier flags a real problem. The cheap rung handles the easy items for near-free; only flagged items pay for the strong model — most of the quality of the big model at a fraction of the cost.

It's the same discipline as the security-audit finder ≠ validator gate, applied to routing: an independent verifier catches the cheap model's blind spots.

The loop

Copy code
1. EXTRACT   cheap model produces an output (extraction / answer / summary)
2. VERIFY    a decomposed verifier asks narrow per-field yes/no questions,
             framed so "bad = true", and returns P(wrong) per question
3. GATE      escalate if ANY question's P(wrong) > fire threshold (max, not mean)
4. ESCALATE  only the flagged items pay for the strong model; the rest keep the cheap answer

Schema-validation is necessary but not sufficient: a cheap model produces confident, schema-valid fabrications (a blank field filled with a plausible invented value). A structural check can't see that — a semantic verifier can.

Verifiers (pluggable) — verify_cascade.py

bashCopy code
VC="$(cat ~/.claude/.bootstrap-dir)/skills/verify-cascade/verify_cascade.py"
python3 "$VC"          # self-test (stubbed judge, no network, no model)

LocalVerifier — the default, nothing leaves your machine

Runs the whole decomposed question battery through a cheap CLI model in one call and returns {field::metric: P(wrong)}. No external service.

pythonCopy code
import sys
sys.path.insert(0, f"{__import__('subprocess').check_output(['cat', __import__('os').path.expanduser('~/.claude/.bootstrap-dir')]).decode().strip()}/skills/verify-cascade")
# (or simply: sys.path.insert(0, "<bootstrap-dir>/skills/verify-cascade"))
from verify_cascade import get_verifier, build_questions, cascade
v = get_verifier("local")                     # uses $MAGGY_JUDGE_CMD (default: deepseek --flash)
result = cascade(
    extract  = lambda: cheap_extract(...),     # your cheap-model call -> dict
    verify   = lambda rec: v.verify({"source_text": source}, build_questions(rec, schema)),
    escalate = lambda: strong_extract(...),    # your strong-model call -> dict
    fire_t   = 0.7,
)
# result: {record, escalated: bool, fired: [qid...], scores: {qid: p}}

Set the judge model with MAGGY_JUDGE_CMD (e.g. qwen3, deepseek --flash) or pass your own judge=callable(prompt)->text. The default passes the prompt on stdin (no shell, so prompt content is never interpreted as a command), so MAGGY_JUDGE_CMD must read from stdin. Questions are scored in complete batches and an over-long source warns on stderr rather than being silently truncated.

TypeSafeVerifier — OPT-IN, and it sends data to a third party

TypeSafe's hosted jev model is purpose-built and calibrated for this. But:

⚠️ Data leaves your machine. TypeSafeVerifier.verify() POSTs the source_text, schema, and extraction to api.typesafe.ai. Only use it for content you are willing to send to a third party. It is off by default — it will not even construct without TYPESAFE_API_KEY, so nothing is sent unless you opt in.

bashCopy code
pip install typesafe_sdk
export TYPESAFE_API_KEY=…            # opt-in; enables the external call

pythonCopy code
v = get_verifier("typesafe")         # raises unless TYPESAFE_API_KEY is set

Everything else (questions, gate, cascade) is identical — the verifier is the only swap.

What makes a good verifier signal

• Narrow + grounded — one checkable yes/no about one field against the source, not a vague "is this good?".
• Bad = TRUE, explicit criteria — frame the escalate case as the true case.
• Per-field, aggregate with max — a per-field flag localizes the error and stays sparse; max means one confident red flag escalates instead of being averaged away.
• Independent + cheap — a dedicated verifier judges the worker's output and catches its blind spots, and it has to be cheap or there are no savings left to capture.
• Calibrated — high on real errors, low on correct ones, so one threshold splits accept vs escalate.

When to reach for it vs plain routing

Use verify-cascade when the output can be checked against a source (extractions, RAG answers, data pulls, structured summaries). For open-ended generation with no ground truth, stick to maggy's task-complexity routing. The two compose: classify to pick the cheap rung, verify to decide whether to escalate.

Attribution

Pattern from TypeSafe's public "SDE cascade" cookbook, rebuilt so the verifier is pluggable and the private LocalVerifier is the default; TypeSafe is an optional adapter.