Imaging-Data Skill
The dataset decides more of a study than the architecture does, and it decides it first. Phases 1–3
establish what the data is and what it will not support, while that is still cheap; Phases 4–7 design
and audit the preparation pipeline so it is leakage-safe before /model-scaffold builds the repo.
Describe-and-audit only: never modify, resample, reorient, split or write image data, never run
preprocessing on real patient data, and wire MONAI / TorchIO transforms by reference rather than
writing a new normalisation or resampling implementation.
Elsewhere: tabular/clinical variables → /generate-codebook, /clean-data; auditing the split
table, held-out metrics, calibration, subgroup results → /model-assessment; choosing an
architecture → /model-selection; building the repo → /model-scaffold.
Workflow
Phase 1 — Profile every case
bashpython3 ${CLAUDE_SKILL_DIR}/scripts/profile_imaging_dataset.py \ --split train:imagesTr:labelsTr \ --split test:imagesTs \ --dataset "MSD Task09 Spleen" \ --declared-labels 0=background,1=spleen \ --target-label 1 \ --plan resample=true,reorient=false,loss=dice_ce,metrics=dice+hd95 \ --out eda/profile.json
One record per case: grid, spacing, orientation, intensity percentiles, the label values actually
present, foreground fraction, and target volume in mL. A --split given no label directory is
recorded as unlabelled — itself a finding. Requires nibabel + numpy; the gate does not.
Every profile figure comes from opening the files — never from a dataset's README, a similar dataset,
or memory. A README can be wrong about its own label indices; the labels cannot.
--target-label on a multi-structure atlas. Foreground defaults to every non-zero index — the
whole annotated anatomy. Measured on the AMOS22 CT cases, that pools to 3.2 % instead of the spleen's 0.20 %, so the
pooled figure sits above the 1 % imbalance threshold while the target sits far below it and the
imbalance verdicts go quiet exactly where the risk is. Naming the target also makes LABEL_EMPTY mean
this case has no spleen. Pass --target-label all for a genuinely multi-class study; leave it out
on a multi-structure atlas and the gate raises TARGET_LABEL_UNDECLARED.
Phase 2 — Gate the profile against the declared plan
bashpython3 ${CLAUDE_SKILL_DIR}/scripts/check_dataset_profile.py --profile eda/profile.json \ --out qc/dataset_profile.json --strict
Stdlib-only, so the audit re-runs anywhere the JSON travels. Never report a profile "pass" without running it.
| Verdict | Severity | Fires when |
|---|---|---|
LABEL_SHAPE_MISMATCH | Major | label grid ≠ image grid |
LABEL_EMPTY | Major | a labelled case has zero foreground |
LABEL_VALUE_UNEXPECTED | Major | label values outside the declared set |
TEST_SET_UNLABELLED | Major | a split named test/held-out/external/eval carries no labels |
ACCURACY_UNDER_IMBALANCE | Major | accuracy is planned while the target is a sliver of the volume |
LABEL_MISSING | Minor | a case in a labelled split has no label file |
SPACING_HETEROGENEOUS | Minor | spacing spans ≥ ratio on an axis and no resampling is declared |
ORIENTATION_MIXED | Minor | >1 orientation code and no reorientation declared |
INTENSITY_SCALE_INCONSISTENT | Minor | some cases on the HU scale, others not |
EXTREME_IMBALANCE | Minor | median foreground below the threshold with no Dice-family loss |
TARGET_LABEL_UNDECLARED | Minor | >1 structure declared, no target named, so foreground pools them all |
The gate flags an undeclared decision, not variability: 5× spacing spread and two orientation
codes pass once resampling and reorientation are declared (the clean challenge fixture proves this).
--spacing-ratio (default 2.0) and --imbalance-frac (default 0.01) are screening defaults, not
published cut-points — never present them as such; the values applied are printed in the output
and belong in the Methods. A split the profile shows unlabelled is never a held-out test set, however
the directory is named.
Phase 3 — Turn the profile into research decisions
Write these decision notes into the study record, so /design-study, the preparation phases below,
and /write-paper inherit them instead of re-deriving them:
- Resampling target — from the spacing distribution, not a tutorial default (carried into Phase 5).
- Loss and metric family — from the foreground fraction. Segmentation reports Dice and a
boundary metric per structure (
/model-assessment); accuracy is not on the list. - Pre-specified subgroups — from the clinical spread the profile shows (target volume, slice thickness, modality). Pre-specifying them here is what separates a subgroup finding from a post-hoc one.
- Where the held-out set comes from — especially when the shipped "test" directory is unlabelled.
- What the cohort cannot support — n, single-source acquisition, absent subgroups: the seed of the Limitations paragraph, written before results can bias it.
Phase 4 — Inventory the preparation steps and fix fit scope
Collect the modality, the data manifest (one row per image/slice with a patient_id), the resample
spacing, the intensity transform (fixed HU window vs a fitted z-score / min-max / histogram match),
and the augmentation plan. Read ${CLAUDE_SKILL_DIR}/references/preprocessing_guide.md for the
modality-aware normalisation, physiology-preserving vs -breaking augmentation, and MONAI / TorchIO wiring.
- Fit dataset-level normalisation on the training split only — never all/full/test.
- Run any data-fitted transform after the split; before it there is no train/test distinction.
- Prefer per-image (per-sample) normalisation where clinically appropriate — leakage-free even before the split.
- Keep augmentation train-only; augmenting val/test folds undisclosed test-time augmentation into the metric.
- Split at the patient level, then map slices to their patient's split.
Phase 5 — Emit the preprocessing manifest
Write preprocessing_manifest.json, which /model-scaffold consumes and the gate checks. Every value
comes from the real data manifest and the declared pipeline — never invented patient IDs or split
assignments.
json{ "split_seed": 42, "transforms": [ {"name": "hu_window", "type": "clip", "fit_scope": "none", "stage": "before_split"}, {"name": "train_zscore", "type": "standardize", "fit_scope": "train", "stage": "after_split"}, {"name": "flip_rotate", "type": "augmentation", "stage": "after_split", "applies_to": ["train"]} ], "split_assignment": [ {"patient_id": "P001", "unit_id": "P001_s1", "split": "train"} ] }
fit_scope: train (OK) · all/full/dataset/test (leak) · sample/per_image/none/fixed
(not data-fitted). stage: before_split / after_split. The fields must describe what the code
actually does — never tag a dataset-fitted transform per-sample to clear the gate; that hides the leak.
A declared dataset-level fit_scope is judged whatever the type is, so a library class name
(HistogramStandardization, NormalizeIntensityd) fit on all is a leak like standardize would be.
Declare the fit scope of resampling too. A target spacing chosen in advance is fit_scope: fixed
and never leaks. A target derived from the cohort does: nnU-Net sets its target spacing from a
percentile of the dataset fingerprint, so a resample fitted over every case carries held-out geometry
into the training grid exactly as an intensity statistic would. The fingerprint's scope decides which
you have, not the word "resample".
Phase 6 — Gate the manifest
bashpython3 ${CLAUDE_SKILL_DIR}/scripts/check_preprocessing_leakage.py --manifest preprocessing_manifest.json \ --out qc/preprocessing_leakage.json --strict
Verdicts: PREPROCESS_BEFORE_SPLIT, NORMALIZATION_LEAKAGE, PATIENT_CROSS_SPLIT (Major);
AUGMENTATION_ON_EVAL, UNSPECIFIED_FIT_SCOPE, MISSING_SEED (Minor), reproduced by set arithmetic
and rule on the manifest. A green gate is the precondition for handing the manifest to
/model-scaffold; its split_assignment is the same patient-level split /model-assessment later
re-verifies. Never report a pass without running it.
Phase 7 — Before inference on a new cohort: check the normaliser's domain
Phase 6 asks whether a transform was fit on the right scope. Before running a trained model on a cohort it was not trained on, ask whether that cohort sits in the intensity domain the trained normaliser assumes:
bashpython3 ${CLAUDE_SKILL_DIR}/scripts/check_normalizer_domain.py \ --profile eda/<cohort>_profile.json \ --contract work/nnUNet_results/.../plans.json \ --splits external_mri --out qc/normalizer_domain.json --strict
Its challenge card holds a cohort in the contract's own domain that must come back clean, an arbitrary-unit cohort that must raise a Major, and an unreadable contract that must refuse rather than pass.
Outputs and hand-off
eda/profile.json(Phase 1),qc/dataset_profile.json(Phase 2), and the decision notes (Phase 3).preprocessing_manifest.jsonwith the augmentation-appropriateness and normalisation fit-scope notes (Phases 4–5),qc/preprocessing_leakage.json(Phase 6),qc/normalizer_domain.json(Phase 7).
The manifest feeds /model-scaffold, which also reads the qc/ reports: keep them in qc/ beside
the manifest (or ../qc/), or pass them with --imaging-qc. An unresolved Major there refuses the
scaffold until it is fixed and re-gated or acknowledged with a stated reason (--ack-qc); Minor and
Flag claims are carried into the repo's IMAGING_QC.md, and a missing report is recorded as not
assessed. Re-run a gate after fixing its finding — a stale report still blocks. The manifest documents the CLAIM 2024 / TRIPOD+AI data-preprocessing items
for /check-reporting; /self-review's model_development probe looks for exactly this pipeline in a
finished manuscript. Regression: bash ${CLAUDE_SKILL_DIR}/scripts/check_dataset_profile_challenge/verify.sh,
bash ${CLAUDE_SKILL_DIR}/scripts/check_preprocessing_leakage_challenge/verify.sh,
bash ${CLAUDE_SKILL_DIR}/scripts/check_normalizer_domain_challenge/verify.sh,
bash ${CLAUDE_SKILL_DIR}/tests/test_dataset_profile.sh,
bash ${CLAUDE_SKILL_DIR}/tests/test_preprocessing_leakage.sh.
