Intuitions
An intuition is a classifier that matches one person's judgment call. Under the hood it is not a fine-tuned model:
- Jev, a judge model, answers each criterion (an observable yes/no or multiple-choice question about the input) and the task itself, zero-shot. Every answer comes back as a probability.
- A small calibrated head learns from the person's labels how much each answer matters. It starts from "trust the zero-shot answer", so 6–10 labels already help.
- An embedding probe learns what the labeled examples look like, beyond what the criteria say.
- Accuracy is always measured on held-out examples the head never trained on.
Three kinds:
output_type | Decision | Example label |
|---|---|---|
choice | exactly one of 2–12 labels | billing |
multi | any subset of 1–8 issues, or none | background+watermark, none |
preference | good/bad, or the better of two | good, a |
Connect
MCP (preferred): streamable HTTP at https://<your stash host>/mcp/intuitions,
with header Authorization: Bearer <Stash API key>.
json{"mcpServers": {"stash-intuitions": {"type": "http", "url": "https://joinstash.ai/mcp/intuitions", "headers": {"Authorization": "Bearer st_..."}}}}
REST: the same bearer token against /api/v1/intuitions and
/api/v1/model-wizard. Classifying one input:
bashcurl -s https://joinstash.ai/api/v1/intuitions/$ID/predict \ -H "Authorization: Bearer $STASH_API_KEY" -H 'Content-Type: application/json' \ -d '{"item": "text, or a JSON object", "ref": "your-id-for-feedback"}'
Build one well
Do these in order. Skip what the person already told you, and never ask more than two questions at a time.
- One decision. What is decided, by whom, and what happens next? If the person describes two decisions, build one now and offer the other next.
- Real inputs first. Ask for one or two real examples before talking about labels.
- Labels with boundaries. Every label needs a definition and the nearest
thing it is not. Ask for the hardest borderline case and what the right call
is; this is where most classifiers go wrong. If several things can be true of
one input at once, use
multi. - Stakes. Which mistake is worse? Which cases must it never get wrong?
Add those as labeled,
pinnedexamples: no version can ship if it gets a pinned case wrong. - Brief. Record all of the above with
save_brief. It is compiled into the instructions the judge reads, so keep it accurate. - Examples. Get 8+ confirmed labels with every label present, mostly real
and hard. Real data first: ask where real cases live (exports, logs,
tickets, past decisions). Only if the person can't provide them, or their
data lacks a kind of case, use
generate_examplesaimed at that gap. Drafts come with a suggested label and reason, and a person must confirm them (label_example, oraccept_suggestionsonly when they say so). Never invent labels yourself. - Criteria. 3–6 observable properties. Once both kinds of label exist,
propose_criteriawith modecontrastivefinds what separates similar items that were labeled differently. Save them withset_criteria. - Train, then report held-out accuracy with its sample size, next to zero-shot Jev. With fewer than about 20 held-out examples, call it an early estimate.
- Ship when the gate passes.
Use it
| Use | How |
|---|---|
| Verify inside an agent loop | classify the draft before acting; retry or escalate on a bad label or low confidence. |
| Monitor production | classify real traffic with a ref. Low-confidence calls land in the inbox. |
| Regression-test evals | Pin must-never cases. Training reports what each new version fixed and broke. |
| Prompt optimization | Use P(good), or a label's probability, as the score to maximize. |
| Post-training reward | The same scores as a reward. Calibrated probabilities make it a smooth signal. |
Keep it improving
inboxlists recent decisions, least confident first.review_predictionconfirms or corrects one, with a reason. It becomes a golden example and can be pinned.give_feedbackrecords downstream outcomes against aprediction_idor your ownref.- Reasons from corrections drive new criteria;
trainagain after a batch of corrections.
MCP tools
list_intuitions, create_intuition, get_intuition, update_intuition, save_brief,
add_examples, generate_examples, list_examples, label_example,
accept_suggestions, propose_criteria, set_criteria, train, ship,
classify, inbox, review_prediction, give_feedback.
