Deterministic Hash Partitioning
Overview
When a dataset is too large to fit in memory (e.g., 50M doodles across 340 classes), deterministic hash partitioning splits it into N shards using key_id % N. Each shard gets a balanced mix of all classes, is reproducible without storing the split, and can be processed independently. Combined with streaming append, this builds sharded files without loading the full dataset.
Quick Start
pythonimport pandas as pd import numpy as np from tqdm import tqdm N_SHARDS = 100 categories = [...] # list of 340 class names for class_idx, category in enumerate(tqdm(categories)): df = pd.read_csv(f"train_{category}.csv", nrows=30000) df["label"] = class_idx df["shard"] = (df["key_id"] // 10**7) % N_SHARDS for k in range(N_SHARDS): chunk = df[df["shard"] == k].drop(["key_id", "shard"], axis=1) mode = "w" if class_idx == 0 else "a" header = class_idx == 0 chunk.to_csv(f"train_shard_{k}.csv.gz", mode=mode, header=header, index=False, compression="gzip")
Workflow
- For each class file, read a fixed number of rows (balanced sampling)
- Compute shard assignment:
key_id // 10^7 % N_SHARDS - Append each shard's rows to the corresponding output file
- After all classes are processed, each shard contains a balanced mix
- Shuffle within each shard (add random column, sort, drop)
- Use shards as independent training chunks for generators
Key Decisions
- Hash function: integer modulo is fast and deterministic; use
// 10^7to avoid sequential correlation - N shards: 100 gives ~300 rows per class per shard with 30K samples/class — small enough for chunked reading
- Compression: gzip each shard to reduce disk I/O (3-5x compression on CSV)
- Shuffle within shard: essential — append order groups by class; random sort interleaves them
- Reproducibility: same key_id always maps to same shard, no random seed dependency
