Decision head training and label supply
When to use
Every off-the-shelf candidate fails the utility gate on your question.
Procedure
- Establish the label supply first: real inputs, existing decisions, weak supervision, or active labelling.
- Freeze a held-out set before training anything.
- Prefer the cheapest adaptation that can work: freeze the backbone and train a small head on a few examples per class.
- Fine-tune on your own question and option shape, since the base checkpoint may be far weaker than a task-fine-tuned one.
- Re-run the frozen exam and compare against the off-the-shelf candidates on the same basis.
Decision rules
- Labels are the constraint, not compute. If you cannot supply labels, stop and choose a different design.
- Never train on the held-out exam set. Its value is that it has never been trained on.
- Compare a fine-tuned small model against the encoder you would otherwise train, on measured cost and agreement, including where the comparison crosses over.
- A fine-tuned model needs the same gate discipline as a purchased one.
Decision-output clarification
For a read-only readiness decision, report prerequisites and the permitted adaptation without initiating training. Preserve held-out isolation. When structured output is requested, return the requested complete object with a short reason rather than a reasoning transcript; never imply that assessment performed training or external checks.
Before returning a structured result, check that each field answers the requested proposition, uses the requested types and vocabulary, and agrees with the explanation. Saved evidence is not an action performed by you; a recommended next step is not an executed action.
Pitfalls
- Starting with training before checking label supply.
- Fine-tuning on the benchmark and reporting the benchmark result.
- Assuming fine-tuning beats a better checkpoint that already matches your question shape.
Done
The label source is stated, the held-out set is untouched, and the trained head is compared to off-the-shelf candidates on the identical frozen exam.
