Gradient

ML Engineering Patterns

Overview

GRADIENT — In ML, the gradient is the directional signal that tells you exactly how to improve. When invoked: assesses pipeline stage (data / training / serving / MLOps), loads the relevant pattern file, and applies ML-specific validation — schema checks, drift detection, training-serving skew guards, latency budgets.

Core principle: ML systems have unique failure modes — data drift, training-serving skew, silent degradation. Test data and models, not just code.

Announce at start: "Running GRADIENT for ML-specific patterns."

Entry Point — First 5 Minutes

Copy code
STAGE ASSESSMENT:

"What stage are you at?"
A) Data collection / ingestion
B) Feature engineering / preprocessing
C) Model training / experimentation
D) Model evaluation / validation
E) Model serving / inference
F) Production monitoring / MLOps
G) Debugging an ML failure

Stage → Section mapping:

• A/B → Data Pipeline Testing (see patterns/data-pipeline.md)
• C → Model Training (see patterns/model-training.md)
• D → ML Evaluation (correctness metrics, calibration)
• E → Model Serving TDD (see patterns/model-serving.md)
• F → MLOps Deployment (see patterns/mlops.md)
• G → Run hunter first, then return with evidence

After identifying stage, ask: "What's the primary constraint — accuracy, latency, cost, or reliability?"

Data Pipeline Testing

Load patterns: patterns/data-pipeline.md

Key tests to implement:

1. Schema validation — column set, feature types, label distribution
2. Distribution shift detection — KS test per feature, covariate shift AUC
3. Null / edge case handling — nulls, empty input, extreme values
4. Data leakage check — correlation with future labels, reproducibility

Rule: Write pipeline tests before writing model code.

Model Training Checklist

Load patterns: patterns/model-training.md

Before training complex models:

• Simple baseline implemented (logistic regression / majority class)
• Baseline metrics documented (accuracy, F1, AUC-ROC, latency)
• Hyperparameter search configured (Bayesian preferred; grid only if ≤ 3 params)
• Early stopping rules defined (patience, min_delta, restore_best_weights)
• Checkpoint strategy set (save_best_only, monitor_metric, metadata)
• Ablation study planned (each component justified)

Model Serving TDD

Load patterns: patterns/model-serving.md

Tests required before deployment:

1. Input validation — schema, range, distribution z-score check
2. Latency — p99 within budget, throughput at expected QPS
3. Fallback — model unavailable → default response, timeout → 504
4. Output calibration — confidence bins match actual accuracy ± 5%

MLOps Deployment

Load patterns: patterns/mlops.md

Required components:

1. Model versioning — semver, git commit + data version + metrics in metadata
2. A/B test config — traffic split, primary metric, guardrail metrics, stopping criteria
3. Drift monitoring — PSI for data drift (threshold 0.1), CUSUM for concept drift
4. Rollback triggers — accuracy drop >5%, error rate >1%, latency >2×, PSI >0.25

ML Evaluation Framework

Cost/latency budgets (set before training, enforce in CI):

• p99 latency: 100ms
• cost per 1k requests: $1.00

Red Flags

Never:

• Deploy without baseline comparison
• Skip drift monitoring in production
• Train without checkpointing
• Use test data for training
• Deploy without rollback plan

Always:

• Test data pipelines before model training
• Validate input distribution matches training
• Track model version with data version
• Monitor prediction distribution in production
• Have fallback for model failures

Integration with Superpowers

Final Checklist

• Data pipeline tests pass (schema, distribution, edge cases)
• Baseline model established and documented
• Complex model beats baseline (with ablation study)
• Input validation tests pass
• Output calibration verified
• Latency budget met (p99)
• Fallback behavior tested
• Model versioned with metadata
• Drift monitoring configured
• Rollback triggers defined