Model Selection

Model-Selection Skill

Two questions, in this order. Phases 1–3 answer a literature question with a stable answer: which architecture family suits the task. Phases 4–6 answer a provenance question: which concrete artifact — repository, revision, checkpoint — will be run, and what its numbers may claim. The skill writes decision notes and runs one stdlib gate; it never downloads, runs, trains or benchmarks a model, and it describes archetypes, not a live SOTA leaderboard.

Elsewhere: building the repo → /model-scaffold; profiling data and planning preprocessing → /imaging-data; validation design, metrics, uncertainty, explainability → /model-assessment; documenting a model you built → /model-card; an LLM/MLLM, including benchmark contamination in that setting → /mllm-eval; AI-vs-expert reader study → /design-ai-benchmarking.

Workflow

Phase 1 — Frame the question

State the task (classification / segmentation / detection / synthesis / transfer), the modality and dimensionality (2-D vs 3-D volume), the labelled-data scale (events / structures, not just images), label availability (lots / few / unlabelled pool), and the constraints (class imbalance, small structures, interpretability, deployment compute).

Phase 2 — Walk the decision tree, read the family card

Follow ${CLAUDE_SKILL_DIR}/references/index.md (task → constraints → default pick). It routes to one card; each gives the source paper, core idea, when-to-use, medical-imaging use, reference implementation, and the typical validation setup for that class.

Never recommend an architecture for a modality or data scale it does not suit (a from-scratch ViT on a few hundred images, 2-D slices for a volumetric structure) — the decision-tree constraints exist to prevent exactly that. If asked for "the best" model, say the zoo is a curated archetype map, not a current SOTA ranking.

Phase 3 — Write the architecture decision note

Write decisions/architecture_choice.md: the task, the chosen architecture, its source paper (mandatory — the Methods cite it), the reason against the constraints, the runner-up and why not, and the matching /model-scaffold template. Never invent a benchmark number or paper claim: cite it (verify via /search-lit), or write [VERIFY] and ask. A reference without a /search-lit-confirmed DOI/PMID is marked [UNVERIFIED - NEEDS MANUAL CHECK].

Phase 4 — Write the model dossier for the concrete artifact

When a concrete candidate exists (a GitHub repo, a Hugging Face checkpoint, a paper's released weights), write model_dossier.json recording what is known; leave unknowns unstated, never guessed — the gate turns an absence into a finding.

jsonCopy code
{
  "model": "OrganSeg-3D v2.5.1",
  "source": {"kind": "github", "url": "...", "version": "v2.5.1", "commit": "abc1234"},
  "licence": {"spdx": "Apache-2.0", "verified_from": "LICENSE at commit abc1234"},
  "intended_use": "research",
  "weights": {"pretrained": false},
  "task": {"model": "3d_ct_organ_segmentation", "study": "3d_ct_organ_segmentation"},
  "reported_validation": [{"dataset": "ExampleBench", "metric": "Dice", "source": "J Ex 2021"}],
  "developed_on": ["ExampleBench"],
  "evaluation_arms": [{"name": "external", "dataset": "OtherCohort-2026"}],
  "hardware": {"claimed": "any CUDA GPU", "verified_on": "GTX 1080 Ti", "verified": true}
}

Read each field from the artifact, not from memory:

• Licence — only from the licence file at the pinned revision, and record which file (verified_from). Never from a README badge, a model-card summary, or memory; an unstated licence is LICENCE_UNSTATED, never "probably MIT".
• developed_on — the paper's own account of where the method was built and tuned. It is the field people skip and the one the gate needs: a method that won a challenge was tuned on it.
• hardware.verified — true only after it actually ran. A support matrix is a different claim: a CUDA capability one compiler accepts may still be refused by another in the same stack.

Phase 5 — Gate the dossier

bashCopy code
python3 ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance.py --dossier model_dossier.json \
    --out qc/model_provenance.json --strict

Stdlib-only and network-free: nothing is fetched and no licence resolved online.

The gate flags a relationship, not a reputation: developed_on: ExampleBench passes cleanly as long as no evaluation arm uses ExampleBench (the clean fixture proves this). Dataset names match as token sequences with a small family-alias table — MSD Task09 Spleen matches MSD, MS Cohort 2026 does not — never by substring.

An explicit empty list ("developed_on": []) states "none" and is not a finding; an absent key is. intended_use must be research, commercial or clinical_deployment, and developed_on / weights.trained_on must be lists of strings; anything else is an input error (exit 2), never a pass.

Known limits. LICENCE_INCOMPATIBLE recognises non-commercial / research-only markers in the licence string (NC, non-commercial, research-only, in SPDX or spaced form). A free-text licence that restricts use without those markers (e.g. "academic use only"), or a copyleft licence under closed deployment, is not detected: read the licence file and record the conclusion yourself.

Phase 6 — Turn a Major into a study decision

A BENCHMARK_PROVENANCE_CONFLICT rarely means abandoning the model (it is often the best-engineered option precisely because it was tuned hard). It changes what the arm may claim:

1. Report that arm as a demonstration that the pipeline runs end to end, not as evidence the method works.
2. Put the evidential weight on an arm whose data post-dates the model, and say so with dates.
3. State the conflict in Methods and Limitations rather than leaving a reviewer to find it.

Never report a flagged arm as independent validation. An EVAL_DATA_IN_TRAINING arm is different in kind: it produces a training-set score and cannot be reported as validation at all. Write this arm-by-arm decision into the study record.

Outputs and hand-off

• decisions/architecture_choice.md — the architecture decision note (Phase 3).
• model_dossier.json — the provenance record (Phase 4); qc/model_provenance.json — the audit (Phase 5).
• The arm-by-arm decision (Phase 6).

Carry them to /model-scaffold (instantiate the template), /model-assessment (arm and validation design, what each arm may claim, metrics), /model-card (provenance section), and /write-paper (Methods cite the source paper; Limitations state any conflict). Gate regression: bash ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance_challenge/verify.sh and bash ${CLAUDE_SKILL_DIR}/tests/test_model_provenance.sh.