Grill Jev

Grill-Jev

Generate a context-specific Jev interrogation battery against a plan, spec, design, code artifact, or any topic. Questions are generated from the artifact itself — not from a static list — so they target the actual risks and gaps in what you provide.

When to invoke

• User says "grill this", "validate my plan", "find holes in my spec", "sanity check", "stress test this", "50 questions", "interrogate", "grill-jev"
• Automatically after any planning output. When a plan, spec, or design is produced — before execution begins — pass it through grill-jev. Any structured output with phases, steps, or checklist items qualifies.
• When the user says "does this look right", "approve this", "is this ready", "review my plan"

How it works

1. Read the artifact — the plan, spec, design, or code being interrogated
2. Generate questions — the LLM executing this skill produces up to 50 context-specific Jev questions tailored to the artifact. It writes the JSON battery to a temporary file; no secondary model call or API key is used. Questions use all structured shapes:
   • Noul with {true: {what, examples}, false: {what, examples}} criteria
   • Choice with {what, not_for, examples} per option
   • Score with {summary, signals} per level
   • Noul with array compare instructions when two state paths need comparison
3. Send to Jev — evaluate the questions against the artifact as state. Use Vercel AI Gateway. Too much context is the most common failure: the artifact is resent with every batch, so split the battery by estimated tokens (jev_limits.request_tokens), not by question count, keeping each request under the size target in skills/shared-patterns/jev-production-lessons.md. When the artifact alone passes the target, send the sections each question needs instead of the whole artifact. scripts/grill-jev.py does the splitting: it packs batches by tokens, and when a batch fails it halves and retries down to single questions, so one oversized or malformed question costs only its own answer. A question that still fails is listed as UNANSWERED (unknown, never a pass) and the run exits 2. If a tiny probe also fails, Jev is down and the run stops. The script rejects a malformed battery before sending (for example, Score levels must be a list).
4. Report findings — high-signal answers with suggested actions

Asking good questions

Question quality depends on context and framing. The rules below were validated through 3 iterative Jev loops.

Context inputs that improve questions:

Pass context via --context "audience: SRE, system: stateful payment service, known constraint: cannot have >5min downtime".

Generation rules (Jev-validated):

1. Specific over generic — questions must name specific steps, systems, or claims in the artifact. "Does step 4's migration define a rollback safe to run under live traffic?" beats "does this have rollback?".

2. Mentions trigger deeper scrutiny, not shallower — when the artifact mentions a risk, gap, or uncertainty, generate MORE targeted questions about it. Acknowledgment is not mitigation. "This is risky" without a defined mitigation is itself a finding.

3. Audience weight — use audience context to focus questions. An SRE needs ops questions. A junior dev needs step-clarity questions. A product owner needs outcome questions.

4. Coverage balance — aim for breadth across relevant categories (completeness, feasibility, risk, scope, verification, consistency, reversibility, security, cost and throughput). Do not cluster all questions on one category. Skip a category only when the artifact has nothing that triggers it.

5. Scale by complexity — simple artifact (15-20 questions), medium (25-35), complex (40-50). Hard cap: 50.

Self-calibration loop — if findings feel generic or off-target, use Jev to improve:

1. Run grill-jev — observe which findings feel shallow
2. Ask Jev: "Which questions were not specific to this artifact? What context would have produced better questions?"
3. Feed that context back via --context and re-run

Question categories

Generate questions covering these nine areas, weighted by what the artifact contains:

Visual goals

When the plan or artifact has a visual goal, require a rendered candidate and a reference image in the state instead of a description. If either is missing, that is the first finding. Load references/visual-plans.md for the state, the question set, and the code. Judge screenshots with scripts/grill-jev.py --image (below). Other visual batteries go through jev_transport.evaluate. API and limits: images.md.

Images and screenshots

--image [NAME=]PATH (repeat) and --images-dir DIR (png, jpg, jpeg, webp, named by file stem) add images. The script prepares each (resize, encode) and forces the direct Jev API. It needs TYPESAFE_API_KEY; without it the run exits 2 with one line, and never tries Vercel. --file and --text are optional when images are given.

Grill these screenshots for clarity and misleading panels:

bashCopy code
python3 scripts/grill-jev.py --images-dir shots/ --questions-file /tmp/grill-questions.json \
  --context "audience: on-call SRE; each image is a Grafana dashboard"
python3 scripts/grill-jev.py --image home=home.png --image cpu=cpu.png --questions-file /tmp/grill-questions.json

Each question picks its images in --questions-file. Both forms are optional and combine:

jsonCopy code
{"questions": {
  "cpu_panel_misleading": {"type": "noul", "report_when": "true", "images": ["cpu"],
    "instructions": {"question": "Does the CPU panel imply load the data does not show?"}},
  "home_vs_cpu_units": {"type": "noul", "report_when": "true",
    "instructions": {"question": "Do {img:home} and {img:cpu} label the same unit differently?"}}
}}

• "images": [names] sends those images in state.images (key = name). The field is removed before the question reaches Jev.
• {img:name} in question text references the image inline; each use bills the image again. An unknown name exits 2.
• A question with neither gets every image. If its request would pass about 24,000 tokens, it is not sent: it shows under UNANSWERED (exit 2) with the fix, add "images". Without --questions-file and with 2+ images, the fallback battery is sent once per image (ids id@name) and the cost is printed on stderr.
• Literal {img:x} in --file, --text, or --context is sent as plain text. If a batch fails, earlier answers are still reported and the rest show under UNANSWERED.
• Requests are packed by tokens (jev_limits.IMAGE_REQUEST_TARGET_TOKENS); questions with the same image set share requests. Each request carries only the images its questions use, so an image is billed once per request that carries it. Overlapping sets are not merged. Few images per question keeps cost down.
• Ask one defect kind per question (see building-with-jev/references/images.md). Pair image judgments with measured numbers.

Plans that call Jev or another metered API

A plan can be complete, feasible, and safe and still fail in production because one run spends the provider's per-second limit. Grill it on arithmetic, not just on prose:

0. Check it against the rules. Load skills/shared-patterns/jev-production-lessons.md and turn every unticked pre-ship checklist item into a cost_ Noul with report_when: "false". Name the exact number in the question: "Does the plan keep every request at or under 4k tokens or a measured reliable size?", "Does it send each stage's requests at once, with an instance cap near floor(0.25 × 250,000 / tokens_per_request)?", "Does it set attempts, per-attempt timeout, run deadline, and a retry budget near 4 × requests × failure rate?"
1. Price it first. When the artifact calls Jev, build (or ask for) a JSON of every request one run sends and run python3 scripts/jev-budget-check.py --payload run.json --concurrency C --concurrent-runs N --attempts A, plus --eval-cases N for any eval the plan runs. A fail is a high-signal finding on its own; a warn goes in the report. Put the check's summary in state under budget so battery questions can inspect it.
2. Ask about throughput explicitly. Include Cost & throughput questions (see references/question-battery.md): does the plan state tokens per run and per second against the account's rate limits (250,000 input tokens per second and 1,200 requests per minute on 2026-09-22; the TypeSafe console holds the current numbers)? Does it fan full detail out over every unit, or cascade? Is in-flight concurrency capped? Do retries use jittered exponential backoff with a per-run budget? Is the eval priced and paced? Which errors mean "back off" on the production transport (Vercel AI Gateway reports upstream overload as 503)?
3. Treat "each request fits" as unproven. A plan that shows every request under the per-request limit has not shown the run fits. Look for the per-second number.
4. Diagnoses need measurements. When the artifact explains a failure (an outage, a size cap, a bad payload), ask whether it measured the run's own rate and retry count and tested a small known-good request on the same transport before concluding.

Plans that use Jev's judgments

A plan whose steps call Jev can fail on the questions rather than the volume. Grill it on the model's known edges (skills/meta/building-with-jev/references/question-design.md, Known model edges), as Noul or Choice questions with named evidence:

• Does any Jev question ask for counting, arithmetic, date order, or a magnitude read from a score? Code should.
• Is any question a compound of several judgments, or a "which one" asked as a Noul per candidate? Split it, or use a Choice with none.
• Does a Choice lean on option order, with no reversed-order check for a high-stakes pick?
• Does the plan pin a versioned model ID, or does it use jev-latest, which moves between versions (1.13.0 now, 1.14.0 about 2026-10-21)? A threshold tuned on one version needs a rerun on the next.
• Does it gate on confidence for every answer, where taking the best option would do? Does it treat a Noul near 0.5 as "medium"?
• Does it put untrusted text in state with no criteria naming what counts as evidence?
• Does it depend on images, Box, or long context, which TypeSafe labels non-production, as the only gate?

Question generation prompt

Use this system prompt to generate the question battery:

Copy code
You are generating a Jev question battery to interrogate an artifact.
The artifact is provided as state at key "artifact".

Generate between 20 and 50 questions. Scale the count to the artifact's
complexity — a 3-step bug fix needs fewer questions than a multi-service
migration plan.

For each question, choose the most appropriate Jev primitive:
- Noul: yes/no probability. Use when you want to know whether something
  is true or absent. Always add structured criteria (true/false with
  what + examples) when the boundary is non-obvious.
- Choice: pick one from a known set. Use for risk levels, categories,
  reversibility classifications.
- Score: position on a spectrum. Use for completeness, timeline
  realism, detection speed.

Use array compare instructions when two or more fields in the state
need to be compared side by side. Score levels are situations, 2 to 10 of them;
phrase every Noul so a high value means yes; ask "which one" as a Choice.

Focus questions on the actual content of the artifact. A plan with no
database steps needs no database migration questions. A plan with a
single deploy step needs no multi-service blast radius question.

Output a JSON dict of question_id -> question definition.

Running the battery

bashCopy code
# File artifact
python3 scripts/grill-jev.py --file task_plan.md --mode plan

# Inline text
python3 scripts/grill-jev.py --text "$(cat task_plan.md)" --mode plan

# The executing LLM writes /tmp/grill-questions.json, then Jev evaluates it.
python3 scripts/grill-jev.py --file design.md --questions-file /tmp/grill-questions.json

The executing LLM owns question generation; the script only evaluates a supplied battery through Jev. Without --questions-file, it uses the static fallback battery. The compact guide in references/question-battery.md provides:

• Example shapes for the LLM question generator
• Fallback when generation is unavailable
• Test fixture for unit tests

Output

Copy code
GRILL-JEV FINDINGS — mode: plan — 31 questions (questions file)
============================================================
HIGH SIGNAL (requires attention):
  [completeness/has_success_criteria] noul=0.89 TRUE — no measurable success criteria
    → add explicit success criteria: observable outcomes, not "it works"
  [risk/migration_live_safety] noul=0.84 TRUE — migration runs against live traffic
    → add maintenance window or use online migration tool

CATEGORY SUMMARY:
  completeness  2 findings   risk  1 finding

OVERALL READINESS: 1.4/3 — partially complete; address findings before executing

Primitive shapes — quick reference

pythonCopy code
# Noul with structured criteria
"has_success_criteria": {
    "type": "noul",
    "instructions": {"question": "Does the plan define measurable success criteria?",
                     "inspect": "artifact"},
    "criteria": {
        "true":  {"what": "Specific, observable outcomes are named",
                  "examples": ["all tests pass", "p95 latency < 200ms"]},
        "false": {"what": "Success is vague or absent",
                  "examples": ["it works", "done"]}
    }
}

# Choice with what/not_for/examples
"overall_risk": {
    "type": "choice",
    "instructions": {"question": "What is the overall risk level?",
                     "focus": "Weigh irreversibility, dependency count, blast radius."},
    "criteria": {
        "low":  {"what": "Reversible, few dependencies, limited blast radius",
                 "not_for": "Any step that cannot be undone",
                 "examples": ["adding an optional config flag"]},
        "medium": {"what": "Some irreversibility or cross-system dependencies",
                   "examples": ["schema migration with rollback plan"]},
        "high": {"what": "Irreversible steps, wide blast radius",
                 "examples": ["deleting a table", "replacing auth system"]}
    }
}

# Score with summary/signals levels
"completeness": {
    "type": "score",
    "instructions": {"question": "How complete is this plan?",
                     "note": "Judge whether a competent engineer could execute it without guessing."},
    "criteria": [
        {"summary": "Critically incomplete",
         "signals": ["missing phases", "undefined terms", "no rollback"]},
        {"summary": "Partially complete",
         "signals": ["main path clear", "some steps vague"]},
        {"summary": "Mostly complete",
         "signals": ["all phases named", "minor details missing"]},
        {"summary": "Complete",
         "signals": ["all steps actionable", "success criteria defined"]}
    ]
}

# Noul with array compare instructions
"assumption_vs_reality": {
    "type": "noul",
    "instructions": {
        "question": "Do the plan's assumptions conflict with the known system context?",
        "compare": ["artifact", "context"],
        "focus": "Look for things the plan takes for granted that could be false."
    }
}

Question-authoring guide: references/question-battery.md. Visual goals: references/visual-plans.md.

Integration into planning workflows

Any skill or agent that produces a plan should call grill-jev before declaring it complete:

pythonCopy code
import subprocess

result = subprocess.run(
    ["python3", "scripts/grill-jev.py", "--file", plan_path, "--mode", "plan", "--questions-file", questions_path],
    capture_output=True, text=True
)
print(result.stdout)
# Non-zero exit when HIGH SIGNAL findings exceed threshold