Checkpointing and Tracking
Use this sub-skill when the task involves Accelerate checkpoint save/load, model export, tracker initialization/logging, distributed-safe logging, profiling, state/RNG handling, or memory cleanup around those workflows.
Route First
- For the core forward/backward/optimizer loop, gradient accumulation, prepared dataloaders, and
skip_first_batches, use../training-loop-integration/. - For FSDP, DeepSpeed, Megatron-LM, or backend-specific checkpoint strategy and consolidation details, use
../distributed-training-backends/. - For large-model dispatch/offload checkpoint loading, route to the large-model or big-modeling sub-skill if present.
Fast Workflows
- Save and resume same-script training state with
Accelerator.save_state()andAccelerator.load_state(); seereferences/checkpointing.md. - Configure automatic checkpoint folders and retention with
ProjectConfiguration; seereferences/checkpointing.md. - Register custom checkpointable objects and save/load pre-hooks for extra metadata; see
references/checkpointing.md. - Initialize trackers with
Accelerator(log_with=...),init_trackers(),log(),get_tracker(), andend_training(); seereferences/tracking-and-logging.md. - Use
accelerate.logging.get_logger()for multiprocess-safe logs and ordered rank logs; seereferences/tracking-and-logging.md. - Profile CPU/GPU/XPU sections with
ProfileKwargsandaccelerator.profile(); seereferences/tracking-and-logging.md. - Diagnose common failures using
references/troubleshooting.mdbefore changing training code.
Bundled Helper
- Run
python scripts/checkpoint_tracker_smoke.py --helpfor usage. - Run
python scripts/checkpoint_tracker_smoke.pyto verify a CPU-only tiny save/load/custom-tracker/logging smoke test without any external tracker service.
Key Constraints
save_state()/load_state()are for resuming the same training script shape, not arbitrary model conversion between unrelated scripts.- Register custom objects only if they implement both
state_dict()andload_state_dict(). - For ordinary model artifact export, prefer
accelerator.unwrap_model(),accelerator.get_state_dict(),accelerator.save(), oraccelerator.save_model()rather than treating a training-state checkpoint as a deployment artifact.


