Backtesting Without Peeking

Rolling origins, expanding windows, and the horizon at which skill disappears.

Ch 12 12.5

Explorer

Temporal & Longitudinal

22 mindifficulty 4/5

Controls

Series

Lengthn160

Seasonal periodp12

Trend slopeβ0.050

Seasonal amplitude5

AR(1) persistenceφ0.550

Higher φ means the noise itself is autocorrelated.

Noise sd2

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Forecasters & evaluation

SES smoothingα0.300

Max horizonH12

Training window48

Fixed size for rolling-origin and the k-fold training set.

Origin (fraction of series)0.75

Evaluation scheme

Focus method (error curve)

Series with origin and forecast fan

The vertical line marks the current forecast origin; each colored path is one method's forecast beyond it.

Seasonal naiveDriftSimple exp. smoothingTrend + seasonal regression

Error vs horizon — Trend + seasonal regression

MASE under Rolling origin (fixed window) vs the true rolling-origin backtest.

true rolling-origin MASEMASE = 1

Scheme-reported error vs true out-of-sample error

One-step-ahead (h = 1) RMSE for every method, by evaluation scheme.

Seasonal naiveDriftSimple exp. smoothingTrend + seasonal regression

true rolling-origin RMSERolling origin (fixed window) RMSE

MASE crosses 1 at

never in range

Trend + seasonal regression, true rolling-origin backtest

h = 1 RMSE, true

2.816

strict rolling-origin backtest

h = 1 RMSE, Rolling origin (fixed window)

2.816

same as true when scheme = rolling

Naive-scale (in-sample MAE)

2.917

denominator of MASE

Read the horizon curve

MASE below 1 means the method beats a seasonal-naive baseline; MASE above 1 means it does worse. Trend and regression methods often win at h = 1 but lose their edge as the horizon grows and persistence in the noise (φ) decays — watch where the solid curve crosses the dashed MASE = 1 line.

Predict before you look

You backtest the same forecasting model two ways: a strict rolling-origin backtest, and a random k-fold split of the same data. Which reports lower error?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A simulated series with trend, seasonality, and AR(1)-correlated noise is forecast by four methods — seasonal naive, drift, simple exponential smoothing, and a trend+seasonal regression. Backtesting under a strict rolling-origin (or expanding-window) scheme uses only data available before each forecast origin; a random k-fold scheme instead assigns points to folds without regard to time, letting the fit see values from after the forecast origin. The two schemes are compared directly on the same models and the same data.

Scaled error and the cost of peeking

The mean absolute scaled error at horizon h divides the backtest's mean absolute error by the in-sample mean absolute error of a one-step seasonal-naive forecast. MASE < 1 means the method beats that trivial baseline; MASE = 1 is the break-even line: at longer horizons most methods eventually cross it as genuine predictability decays.

A correct backtest respects the arrow of time: at origin T, only observations with index ≤ T (rolling window of fixed size, or expanding from the start) may enter the fit, and the forecast is evaluated against y(T+h) for h = 1..H. Averaging that error over many origins gives an honest estimate of deployment performance.

A random k-fold split ignores the arrow of time. When a test point at time t has training data drawn from both before and after t, models with a time trend or seasonal term are effectively interpolating between two known anchors rather than extrapolating from the past — which mechanically lowers the reported error without improving the model that will actually run in production, where only the past is ever available.

MASE(h) = MAE(h) / mean_t |y(t) − y(t − period)|

Rolling/expanding origin: fit uses {y(i) : i ≤ T},  test error = y(T+h) − ŷ(T+h)

Random k-fold: fit uses a random subset that may include i > T — the leak

Try this

Leave the scheme on rolling-origin and slide the max horizon out. Watch each method's MASE curve rise and note where it crosses 1.
Switch to random k-fold with the same model and window. Compare the reported RMSE bars to the true rolling-origin bars for every method — which method's advantage is most inflated?
Raise AR(1) persistence φ toward 0.9. Short-horizon MASE improves for real reasons; does the crossing horizon move, and does k-fold's inflation get better or worse?
Shrink the training window toward the seasonal period. Trend+seasonal regression becomes unstable while seasonal naive is unaffected — connect this to degrees of freedom.
Move the origin later in the series (more history available) and compare expanding-window forecasts to rolling-window forecasts at the same origin.

Common misconception

“Cross-validation is cross-validation — a random k-fold split gives the same honest error estimate for a time series as it does for i.i.d. data.”

Random k-fold assumes exchangeability: any observation could equally well be used to predict any other. Time series break that assumption by construction — the future depends on the past, not vice versa. Letting a fold's training set include points after the target time leaks information a real forecaster would never have, and the reported error is optimistic, sometimes dramatically so. Rolling-origin or expanding-window evaluation, which only ever trains on the past, is the honest analogue of cross-validation for sequential data.

§12.5 Forecast validation