Litdata

LitData

LitData (import litdata) preprocesses and streams datasets for PyTorch training:

• Write (optimize / map) → chunked chunk-*.bin + index.json → src/litdata/processing/
• Read (StreamingDataset + StreamingDataLoader) → cache → decode → batch → src/litdata/streaming/
• Raw (StreamingRawDataset) → stream original files without optimize → src/litdata/raw/

To use the library expertly: always load reference/using-litdata.md first. Narrative source: repo README.md.

Install this skill

From any project (Cursor, Claude Code, and other agents supported by the skills CLI):

bashCopy code
npx skills add Lightning-AI/litData

Useful options: -g (user-global), -a cursor (Cursor only), -y (non-interactive). In this repo the skill already lives at .claude/skills/litdata/.

Expert usage (load using-litdata.md)

Before writing examples or answering how-tos, read the cookbook. Highlights:

Reference map

Public API (src/litdata/__init__.py)

Defined under streaming/, processing/, raw/, utilities/ — see cookbook §6–9 for constructor args.

Package map

• streaming/ — read · processing/ — write · raw/ — raw stream · types.py — media wrappers · cli/ — litdata cache path|clear
• utilities/ — env, encryption, subsample, split, parquet, HF
• constants.py — optional-dep flags, env knobs, default chunk 64 MB
• Registries: downloaders, fs providers, serializers, compressors; resolver.py → Dir

Shared concepts

• Chunk: [num_items][offsets][data]; index.json holds chunks + config (data_format, item_loader, …) — storage-format.md.
• Item loaders own layout + intervals (PyTreeLoader, TokensLoader, ParquetLoader).
• Training downloads = Downloader. Optimize remote I/O prefers the same Downloader (adownload_file / aupload_file); FsProvider remains for index merge and fallback. See data-movement.md. Sampler ChunkedIndex is read-path; CacheBatchSampler is CacheDataLoader only.
• Ranks from env (_DistributedEnv / DATA_OPTIMIZER_*), not a custom network.
• Shuffle deterministic from seed+epoch+chunk → resumable (shuffle.py, not sampler.py). Local POSIX-fast keeps FullShuffle unless the mount is Vast/NFS/Lustre/GPFS (or LITDATA_POSIX_FAST=1 → WindowShuffle).
• Design: one less thing to remember; pure PyTorch; backward compatible; test-driven.

Traps (do not “fix” by guessing)

• FileNotFoundError / ChunkWaitTimeoutError after ~120s on chunks with num_workers>0 is usually prefetch-thread death or wait timeout — not a missing object. Check stderr + tracer crash.
• mode="append" vs use_checkpoint=True: append continues chunk indices; checkpoint resumes input slices. Combining them is unsafe; checkpoint filenames / done_till_index are known-fragile (writer.save_checkpoint vs data_processor load).
• Azure/HF resolve for read; optimize write providers are s3/gs/r2 only (_SUPPORTED_PROVIDERS).
• dataset_update: local dirs only; ds[int] is positional.
• Empty Tensor() is not tokens. Only 1-D Tensor(array=…) (or path= / bytes= + dtype=) uses the TokensLoader layout. Empty Text() / JpegArray() fail on write.
• Bare .txt / .npy / .bin path strings pickle the path. Wrap with Text / File / load the array. Bare *.jpg / *.png / *.wav paths are claimed as media.
• litdata_collate only batches graphs. A dict with AudioDecoder / video decoder + a stackable field needs a custom collate_fn (see examples/modality/audio.py).

Quick commands

bashCopy code
make setup
pre-commit run --all-files
mypy
pytest tests/path/test_x.py::test_name -v --capture=no
litdata cache path
litdata cache clear

Examples: examples/ (modality recipes: examples/modality/). Version: src/litdata/__about__.py.