Model-Scaffold Skill
Purpose
This skill stamps out a runnable PyTorch training repo for a medical-imaging task — --task
segmentation (U-Net), classification (CNN / timm backbone), detection (torchvision Faster
R-CNN / FPN), synthesis (Pix2Pix generator + PatchGAN), ssl (SimCLR encoder), or finetune
(transfer-learning a pretrained backbone with a frozen→unfrozen schedule + a provenance record) —
with the reproducibility guarantees baked in by construction — so the build is leakage-safe and
reproducible before a single epoch runs. It is the imaging analogue of how /analyze-stats generates
runnable statistical code: the generator produces the repo, you run the training on your GPU / Colab,
and the lane's deterministic gates verify the network-free parts.
It is the missing middle link in the lane: /model-selection (choose) → model-scaffold (build)
→ /model-assessment (validate the split / design, compute metrics) → /analyze-stats
→ /write-paper + /check-reporting (publish). It integrates MONAI / nnU-Net / TorchIO (referenced
in the generated requirements.txt); it does not reimplement them.
When to use
- You have a data manifest (one row per image, with a patient/subject ID) and want a reproducible, leakage-safe starting repo for a segmentation, classification, detection, synthesis or self-supervised model.
- You want to fine-tune a pretrained backbone (transfer learning — the common clinician workflow:
a
timm/ MONAI / MedSAM checkpoint adapted to your collected clinical data) with the freeze schedule, discriminative learning rates, and pretrained-weight provenance recorded (--task finetune).
When NOT to use
- Auditing an already-trained model's validation design →
/model-assessment. - Held-out metrics / calibration / bootstrap CIs →
/model-assessmentthen/analyze-stats. - Choosing the architecture for the research question →
/model-selection. - Reimplementing MONAI / nnU-Net → out of scope (the scaffold integrates them).
- LLM / MLLM evaluation →
/mllm-eval.
Workflow
Phase 1 — Prepare the manifest
A CSV with one row per image and a patient/subject ID column (patient_id / subject_id /
case_id), plus image and label path columns. The ID column is load-bearing: the split is done at the
patient level off this column. IDs are compared after stripping surrounding whitespace (P01 and
P01 are one patient), in the split and in the generated dataset.py alike.
Phase 2 — Generate the repo
bashpython3 ${CLAUDE_SKILL_DIR}/scripts/scaffold.py \ --manifest <manifest.csv> --task segmentation --out model_repo --seed 42 \ --in-channels 1 --out-channels 1 # --task = segmentation | classification | detection | synthesis | ssl | finetune # (out-channels = num classes for classification/finetune, target channels for synthesis; # finetune uses a softmax CrossEntropy head, so it refuses --out-channels < 2 — binary = 2) # fine-tuning a pretrained backbone (transfer learning) on collected clinical data: python3 ${CLAUDE_SKILL_DIR}/scripts/scaffold.py \ --manifest <manifest.csv> --task finetune --out model_repo --seed 42 \ --out-channels <num_classes> --from-pretrained timm:resnet50.a1_in1k # emits PRETRAINED.md (provenance) + a frozen→unfrozen train.py with discriminative LRs; # record the exact pretrained source so the fine-tune is reproducible. build_model(pretrained=True) # raises if timm is missing (never a silent random-init stand-in); best.pt records backbone_class. # reuse the split /imaging-data's preprocessing gate checked (do not draw a new one): python3 ${CLAUDE_SKILL_DIR}/scripts/scaffold.py \ --manifest <manifest.csv> --preprocessing-manifest preprocessing_manifest.json --out model_repo # copies its split_assignment + split_seed into splits/, reading rows with the gate's own rules # (patient key patient_id/subject_id/patient/id; split synonyms such as training/validation/holdout); # exits 2 if a manifest patient has no split there, a patient sits in two splits, a split does not # map to train/val/test, or split_seed is absent.
The imaging-data QC handoff is enforced, not advisory. With --preprocessing-manifest the
scaffold also reads /imaging-data's gate reports (check_dataset_profile,
check_preprocessing_leakage, check_normalizer_domain JSON) from <manifest dir>/qc/ and
<manifest dir>/../qc/ (the manifest path is resolved first). --imaging-qc <file|dir> (repeatable)
replaces that search: pointed at an empty directory, every gate is recorded NOT ASSESSED. A
leakage report is skipped only when its recorded manifest has a different file name, resolves to an
existing file, and no candidate path holds a byte-identical copy of the scaffolded manifest (the
reason is shown); anything else — a copied project, a same-named manifest elsewhere — is read.
- A Major claim — or any severity that is not plainly Minor/Flag, or a report whose
summary.n_majorexceeds its listed Majors — refuses: exit 1, nothing written, each code + report listed. Resolve it upstream and re-run the gate, or pass--ack-qc CODE='reason'once per code. - Minor / Flag claims never block; they are carried forward as warnings.
- A gate with no readable report is NOT ASSESSED; an unparseable or off-shape file in
qc/is listed as UNREADABLE (stderr + record), never dropped.
All of it lands in model_repo/IMAGING_QC.md, referenced from config.yaml (imaging_qc:) and
REPRODUCIBILITY.md, so training, evaluation and the Methods read what /imaging-data found. Never
write an --ack-qc reason the user has not given. Without either flag the output is unchanged.
This writes model_repo/ with config.yaml, model.py (the task's model — U-Net / CNN / Faster R-CNN
/ Pix2Pix / SimCLR encoder), dataset.py (reads the frozen split), losses.py (task-appropriate),
train.py, evaluate.py, requirements.txt,
REPRODUCIBILITY.md, methods_stub.md (+ IMAGING_QC.md when imaging-data outputs are given), and — the key artifact — splits/split_assignment.csv +
splits/split_seed.txt. The split is patient-disjoint by construction (a deterministic group split)
and the emitted code seeds every RNG, sets cuDNN deterministic, builds the training loader from the
train split only, and infers under model.eval() + torch.no_grad().
Phase 3 — Verify the build (network-free)
bash# this skill's own training-hygiene gate python3 ${CLAUDE_SKILL_DIR}/scripts/check_training_hygiene.py --repo model_repo --strict # the split-leakage gate (proves patient disjointness) — owned by /model-assessment
Route the emitted splits/split_assignment.csv to /model-assessment
(check_split_leakage.py --splits model_repo/splits/split_assignment.csv --strict) for the
patient-disjointness proof, and (optionally, locally with torch installed)
bash ${CLAUDE_SKILL_DIR}/scripts/scaffold_challenge/verify.sh to smoke the forward pass.
Phase 4 — Plug in your data and train
Implement dataset.py's _load_image / _load_label for your modality (DICOM / NIfTI / TIFF via
nibabel / pydicom / tifffile / TorchIO / MONAI transforms). For production, swap model.py for MONAI
UNet / SegResNet or an nnU-Net plan (see ${CLAUDE_SKILL_DIR}/references/training_guide.md). For a
fine-tuning repo (--task finetune), fill PRETRAINED.md and set the freeze schedule / discriminative
learning rates (see ${CLAUDE_SKILL_DIR}/references/finetuning_guide.md, which also covers MedSAM/SAM
adaptation and train-only diffusion augmentation). Run python train.py (best model selected on the
val split), then python evaluate.py (predictions on the test split, touched once).
Phase 5 — Validate, evaluate, publish
Hand off to /model-assessment (validation-tier + comparator + metric-selection audit; Dice +
HD95/NSD with CIs) + /analyze-stats, /make-figures, and /write-paper
(fill the methods_stub.md [VERIFY] placeholders) + /check-reporting (CLAIM 2024 / TRIPOD+AI). For
reproducibility-safe wiring of experiment tracking (W&B / MLflow), config / data / environment
versioning, and the MLOps reporting checklist, see ${CLAUDE_SKILL_DIR}/references/mlops_guide.md
(a wiring + reporting reference — it points to the frameworks, it does not replace them).
Runnability — honest contract
The generated repo is runnable, but runnability is not a CI guarantee. The default gates prove
the network-free properties (the emitted split is patient-disjoint + seeded; the emitted training code
is hygienic) by parsing the produced artifacts — no torch is executed. A torch forward-pass smoke
(build + forward shape + gradients flow + reproducible loss) is a self-skipping tier in the
challenge verify.sh and a documented local command; it is never counted as CI coverage of
runnability.
Anti-Hallucination
- Never fabricate training or evaluation metrics. The scaffold emits
[VERIFY]placeholders; every number must come from the user's executed run and from/model-assessment+/analyze-stats. - Never emit a split that is not patient-disjoint or not seed-locked. The generator does this by construction; do not hand-edit the split table to introduce overlap or remove the seed.
- Never claim the generated repo was trained or that it achieved a result — it is a starting point the user runs.
- If a library API, default, or architecture detail is uncertain, flag
[VERIFY]and ask rather than guessing.
Deterministic gates
scripts/scaffold.py— the generator (stdlib + numpy; deterministic given manifest + seed); also the imaging-data QC handoff (tests/test_imaging_qc_handoff.sh).scripts/check_training_hygiene.py— AST linter: all RNGs seeded, cuDNN deterministic,eval()+no_grad()inference, no training on a non-train split, and (fine-tuning) a recorded pretrained-weight provenance when pretrained weights are loaded (PRETRAINED_PROVENANCE_MISSING).scripts/scaffold_challenge/verify.sh— the build → validate chain, network-free (torch tier self-skips).
Known limits of check_training_hygiene.py
- It counts seeding / cuDNN /
eval()/no_grad()only in code a run reaches from the script's import-time statements. A file with no import-time call into its own functions is entered through every public top-level function nothing references, so an uncalled publicseed_everythingin such a file still counts. Every decorated function or method (a click/typer command, a route, a@staticmethod) is also an entry point, so an uncalled decorated seeding helper still counts. - It does not check statement order: inference placed before
model.eval()in the same function is not detected. --repowithouttrain.py/evaluate.pyreports those checks as NOT CHECKED; with--strictit exits 2 rather than clearing them.- Dataset variables are resolved by one name-to-split map for the whole file (later assignment in source order wins, across functions), not per scope; a shuffled loader that combines train with val only (a train+val refit) is not flagged.
Boundaries
model-selection (choose) └─ model-scaffold (this skill: generate the reproducible repo) ├─ check_training_hygiene.py (training-code hygiene) ├─ model-assessment -> analyze-stats (split-leakage proof, validation design, metrics + CIs) └─ write-paper + check-reporting (Methods stub -> compliant manuscript)
