Forecasting

Forecasting

A forecast that cannot beat "repeat last period" is noise. Baseline first, fancy second. The naive forecast is free, instant, and the bar every model must clear — if your AutoARIMA loses to last-quarter-repeated, ship the repeat and say so.

You are not done when a model produces a number. You are done when you can defend the number: which method, why that method for this data, how it scored against the naive baseline in a backtest, and the interval around the point. A point estimate with no error band is a guess wearing a lab coat.

The deliverable contract

Every forecast you ship is a reproducible artifact, not a number pasted in chat:

1. A script that reads the history and regenerates the forecast (no manual steps).
2. A CSV/Parquet with columns ds, forecast, lo, hi — timestamp, point, interval bounds.
3. A one-paragraph accuracy readout: WAPE + bias from a rolling-origin backtest, and MASE vs the naive baseline (MASE < 1.0 = you beat naive; ≥ 1.0 = ship the naive forecast instead).

If you cannot produce all three, you have not forecast — you have guessed. scripts/verify.sh checks the artifact has these columns, the right row count, and an accuracy line.

The loop

Run these in order. Skipping step 3 is the most common failure.

1. Frame it. Pin down the horizon h (how many periods forward), the granularity (daily / weekly / monthly), and exactly what is being predicted (units? revenue? per-SKU or aggregate?). Forecast at the level you will act on — if you reorder per SKU, forecast per SKU, then sanity-check against the aggregate.
2. Establish the series. Regular timestamps, one row per period, gaps filled explicitly (a missing month is not zero unless it truly is). Flag promotions, stockouts, and outliers — they distort the signal. If the input is dirty (dupes, missing rows, mixed units), stop and hand off to data-cleaning before modeling. Garbage history, garbage forecast.
3. Build the naive + seasonal-naive baseline. This is the bar. Naive = repeat last value. Seasonal-naive = repeat the value from one season ago (e.g. last December for this December). Compute its backtest error now — every fancier method must beat it or lose.
4. Pick the method by data shape (table below). Do not reach for ARIMA on instinct.
5. Backtest with rolling-origin cross-validation. Never a single holdout. Compute WAPE + bias + MASE vs the naive baseline across multiple cutoffs.
6. Report. Point + interval, the one-line method rationale, the accuracy readout. Then hand off downstream (inventory, financial-model).

Method selection

Match the method to the shape of the history, not to what sounds sophisticated. statsforecast (Nixtla, v2.0.3) provides all of these with built-in intervals.

When in doubt between two, fit both plus the baseline in one StatsForecast run and let the backtest decide. Theta (AutoTheta) is a strong, cheap default that often wins on monthly business series.

Accuracy and honesty

The metrics are not decoration — they decide what you ship.

• WAPE, not MAPE. MAPE divides by the actual, so it explodes and misleads whenever actuals approach zero (constant in SKU and intermittent data). WAPE = total absolute error / total actual volume — volume-weighted and stable. It is the default magnitude metric.
• Pair WAPE with bias. WAPE is how big the error is; bias is the direction — whether you systematically over- or under-forecast. A 10% WAPE with +9% bias means you are almost always forecasting high, an actionable problem quite different from random error.
• MASE < 1.0 is the pass/fail line. MASE is scale-free: your error over the naive forecast's error. Below 1.0 you beat naive; at or above it, ship the naive forecast instead. The single most important number in the readout.
• Rolling-origin, never a single holdout. Time-series CV repeats the train/test split across multiple cutoffs (expanding window), a far more reliable estimate than one lucky/unlucky split. Use cross_validation(h=…, n_windows=…).
• Always emit an interval. A point forecast cannot express uncertainty, and point metrics cannot evaluate a distribution. Report level=[80] or [95]. The interval is half the deliverable, not optional polish.

Formulas (WAPE, MASE, bias, pinball, coverage), rolling-origin mechanics, and how to read a backtest table are in references/accuracy-and-backtesting.md. Per-method when-to-use and the exact statsforecast one-liner for each are in references/methods-cheatsheet.md.

Minimal pipeline

The full pipeline: long-format dataframe, fit competing methods + baseline, backtest, forecast with an interval, write the artifact.

pythonCopy code
# pip install statsforecast  (Nixtla, v2.0.3)
import pandas as pd
from statsforecast import StatsForecast
from statsforecast.models import SeasonalNaive, AutoETS, AutoARIMA

# long format: unique_id, ds, y  (one row per series per period)
df = pd.read_csv("history.csv", parse_dates=["ds"])
m, h = 12, 12  # monthly seasonality; forecast 12 periods ahead

sf = StatsForecast(
    models=[SeasonalNaive(season_length=m), AutoETS(season_length=m), AutoARIMA(season_length=m)],
    freq="MS",
)

# rolling-origin backtest BEFORE trusting any forecast
cv = sf.cross_validation(df=df, h=h, n_windows=3, step_size=h)
def wape(a, f): return (a - f).abs().sum() / a.abs().sum()
for col in ["SeasonalNaive", "AutoETS", "AutoARIMA"]:
    print(col, "WAPE", round(wape(cv["y"], cv[col]), 4))  # pick the lowest that beats SeasonalNaive

# refit on full history, forecast with an 80% interval
fc = sf.forecast(df=df, h=h, level=[80])
# choose the winning model column from the backtest; here AutoETS as example
out = fc.rename(columns={"AutoETS": "forecast", "AutoETS-lo-80": "lo", "AutoETS-hi-80": "hi"})
out[["ds", "forecast", "lo", "hi"]].to_csv("forecast.csv", index=False)

Zero-dependency fallback when you cannot install statsforecast — a seasonal-naive baseline in pure pandas. This is also the thing every model must beat, so it is always worth computing:

pythonCopy code
import pandas as pd

def seasonal_naive(y: pd.Series, m: int, h: int) -> pd.Series:
    """Repeat the last full season forward h periods."""
    last_season = y.iloc[-m:].to_numpy()
    return pd.Series([last_season[i % m] for i in range(h)])

s = pd.read_csv("history.csv", parse_dates=["ds"]).set_index("ds")["y"]
fc = seasonal_naive(s, m=12, h=12)
# crude interval from historical residual spread; honest is better than absent
resid_std = (s - s.shift(12)).dropna().std()
out = pd.DataFrame({"forecast": fc, "lo": fc - 1.28 * resid_std, "hi": fc + 1.28 * resid_std})
out.to_csv("forecast.csv", index=False)

Edge cases

• New product, no history. Do not fit a model on three points. Use analogues — a comparable product's curve scaled to expected volume — and say it is an assumption, not a forecast.
• Short history (< 2 seasonal cycles). SeasonalNaive only. A fitted model will overfit noise and report a falsely tight interval.
• Structural breaks. A pricing change, a relaunch, a regime shift. Do not train across the break — train on the post-break segment, even if it is short, or the model averages two different worlds.
• Promotions and outliers. A promo spike is not baseline demand. Mark promo periods and either model them as a regressor or exclude them from the level estimate; otherwise the forecast inherits a spike that will not recur.
• Granularity. Forecast at the level you act on. If you must report higher, aggregate the forecasts — and check the aggregate is plausible (this often catches per-SKU nonsense).
• Hierarchy reconciliation. When SKU forecasts must sum to a category total, reconcile (bottom-up or MinT). Brief note here; the mechanics belong in references/methods-cheatsheet.md.

Anti-patterns

Handoffs

• ../inventory/SKILL.md — feed it the demand number; it sizes reorder points and safety stock. Forecasting produces the demand; it does not size the stock.
• ../financial-model/SKILL.md — when the projection is driven by assumptions and drivers (pricing, hiring), not history, that is a model, not a forecast.
• ../unit-economics/SKILL.md — contribution margin, CAC/LTV, payback. No time series, route there.
• ../data-cleaning/SKILL.md — dirty input (dupes, missing rows, mixed units) goes here before you model.
• ../analyze/SKILL.md — when the question is "why" or a backward-looking metric/aggregation rather than forward extrapolation.