Data Pipelines, Scheduled Jobs and Data Integrity
Data integrity: what users need is restores, not backups
Users judge you by whether their data is there and correct when they ask for it. A backup that has never been restored is a hope, not a plan.
Threat model
Data loss comes from many combinations of:
- Root cause: user error, application bugs, operator/automation error, infrastructure failure, malicious action, migration bugs.
- Scope: wide (many entities) vs. narrow (one user/object).
- Rate: sudden (big bang) vs. slow and creeping (a bug corrupting a little data each day for months).
Slow, creeping corruption is the most dangerous: by the time it's noticed, it's in every recent backup and replica.
Defense in depth
- First layer — soft deletion / undo: deleted data goes to a trash/tombstone state for a retention period (days to weeks) before purge; users and support can undo. Protects against user and application errors.
- Second layer — backups and recovery: versioned backups with retention long enough to catch slow corruption, stored in a separate failure domain and administrative domain (so the same bug or operator mistake can't delete both). Replication is not a backup — it faithfully replicates corruption and deletes.
- Third layer — early detection: continuous out-of-band validation that checks data invariants (referential integrity, counts, checksums, business rules) and alerts on drift. Detection time is what bounds how far back you need to restore.
Recovery
- Define RPO (how much data loss is tolerable) and RTO (how fast you must recover) per dataset, and design backups to meet them.
- Test restores continuously and automatically, end to end, from real backups; measure time to restore. Alert when a restore test fails.
- Support partial/point restores (one user, one table, one time range), not just "restore everything".
- Keep tooling and runbooks for recovery as production-grade as serving code.
Data pipelines
Design
- Prefer continuous/streaming or frequent small batches over huge periodic batches when freshness matters; big periodic jobs create load spikes ("thundering herds") and long recovery after failure.
- Make every stage idempotent and re-runnable; write outputs atomically (write then rename/commit).
- Define pipeline SLOs: freshness, correctness, coverage/completeness, and end-to-end latency (
sre-slos-and-error-budgets). - Handle stragglers and hot spots: split work into many small shards, rebalance dynamically, avoid skewed keys.
- Schema changes and migrations: versioned schemas, backward/forward compatible readers, dual-writes and backfills staged and verified.
- Isolate pipelines by priority so a large low-priority backfill can't starve a critical job.
- Plan dependencies: know upstream sources, their SLOs, and what happens when they are late or bad; validate inputs and quarantine bad records instead of failing whole runs or silently passing garbage.
Operating
- Monitor: freshness/lag at each stage, throughput, backlog, error/quarantine counts, data quality checks.
- Alert on SLO (e.g. "data older than X for > Y% of reads"), not on every task failure.
- Keep runbooks for: reprocessing a time range, skipping/quarantining poison inputs, backfills, and rolling back a bad output version.
- Canary pipeline code and config changes on a subset of data and compare output against the previous version before full rollout.
Distributed cron / scheduled jobs
- Run the scheduler itself redundantly with leader election via a consensus system, so one machine's failure doesn't skip or double-run jobs.
- Persist launch state durably: record intent before launching and success after, so a new leader can tell whether a job ran.
- Make jobs idempotent or have them check for prior completion; decide explicitly whether a missed run should be skipped or run late, and whether a double run is acceptable.
- Spread start times (jitter, hashed minutes) to avoid everyone running at :00 and overloading shared systems.
- Monitor for missed runs, overruns (job still running at next start), and failures.
Review checklist
- RPO/RTO defined per dataset; backups meet them?
- Backups in separate failure + administrative domain; retention covers slow corruption?
- Soft-delete / undo for user-facing deletions?
- Automated, end-to-end restore tests with alerting?
- Out-of-band data validation running?
- Pipeline stages idempotent, atomic outputs, freshness/correctness SLOs?
- Scheduled jobs survive scheduler failure without skipping or double-running?
Sources
- SRE book ch. 24 Distributed Periodic Scheduling with Cron — https://sre.google/sre-book/distributed-periodic-scheduling/
- SRE book ch. 25 Data Processing Pipelines — https://sre.google/sre-book/data-processing-pipelines/
- SRE book ch. 26 Data Integrity — https://sre.google/sre-book/data-integrity/
- Workbook ch. 13 Data Processing Pipelines — https://sre.google/workbook/data-processing/
