Walk-forward analysis

An edge that only exists on the data you fitted it to is not an edge — it is a memory. Walk-forward analysis is the discipline of always testing on data the strategy has never seen, in the order it actually arrived. This chapter covers in-sample vs out-of-sample, the danger of tuning on everything, rolling re-optimisation, and why purging and embargoing are non-negotiable when labels overlap.

The intuition is the one every teacher knows: if you let students see the exam before it counts, everyone scores 100 and the grade means nothing. A strategy tuned on the same data you then report is a student who already saw the exam. Walk-forward is the discipline of keeping a sealed exam — data the strategy has never touched — and grading only on that, in the order time actually delivered it. The number that survives is the one you can expect to repeat live; the tuned-on-everything number is the leaked exam.

In-sample vs out-of-sample#

Split your history at a date. Everything before is in-sample (IS) — the data you are allowed to look at, tune on, stare at. Everything after is out-of-sample (OOS) — data you touch exactly once, to measure. The moment you use the OOS block to make a decision, it becomes in-sample, and you have no clean test left.

in-sample (tune here) r_{1}, \dots, r_{k} out-of-sample (measure here, once) r_{k + 1}, \dots, r_{T}

edgekit gives you two cuts. oos_split makes a hard calendar cut into date-indexed IS/OOS slices; is_oos_splitmakes a positional chronological cut at a fraction — no shuffle, preserving time's arrow.

oos.py

import edgekit as ek
import numpy as np

# calendar cut — measure IS -> OOS Sharpe degradation
in_sample, out_of_sample = ek.validation.oos_split(df, cut="2023-01-01")

# positional cut at a fraction (chronological, no shuffle)
a, b = ek.validation.is_oos_split(np.arange(100), frac=0.55)   # len 55 / 45

✦Scenario — the edge that halved out of sample

You tune a Donchian breakout on BTCUSDT over 2019–2022: you sweep the channel length, land on 55 bars, and it prints Sharpe 1.4 in that window. Feels like an edge. Now you run it — untouched — on the sealed 2023–2025 block. Sharpe comes in at 0.7. Half the edge evaporated. That is not necessarily failure: some decay is expected (2019–2022 included the choice of "55", so it was always flattered). The question is how muchdecayed. A drop from 1.4 to 0.7 is survivable edge-decay you can haircut for; a drop from 1.4 to 0.1 is a leaked-exam collapse — you fitted 2019–2022's noise and there was never an edge to carry forward. Same test, and the size of the gap is the whole verdict.

!Never shuffle a time series before splitting

Random train/test splits are correct for i.i.d. data and catastrophic for time series. Shuffling lets the model train on the future and test on the past — a leak that guarantees an OOS number you can never reproduce live. Splits must respect the arrow of time.

The danger of tuning on all the data#

Suppose you sweep a parameter over the whole history and pick the value with the best Sharpe. That Sharpe is now an in-sample number for the entire dataset — you have no data left that the choice did not see. This is the most seductive form of overfitting because it does not feel like cheating: you ran one honest backtest per value. But the selection used all the data, so the winner is partly fitted to noise, and its forward performance will disappoint by the amount of that noise.

The only cure is to make the selection itself out-of-sample: choose parameters on the past and measure on a future the choice could not have seen. That is walk-forward.

Walk-forward optimisation#

Walk-forward slides a train/test window through time. On each fold, fit or select the configuration on the training window, then apply it — unchanged — to the immediately following test window. Concatenate the test blocks and you have a fully out-of-sample track record built the way you would actually have traded it: decide on the past, live with it on the future, roll forward.

Rolling train and test windows advancing through time, each test block following its train block — Walk-forward: each fold fits on its train window and is measured on the next, unseen block. The test blocks concatenate into an honest OOS record.

edgekit's walk_forward has two modes, selected by refit:

refit=False (the honest default) — split one strategy's returns into k consecutive blocks and check each is positive. An edge should be spread across time, not concentrated in one lucky block.
refit=True — expanding-window re-optimisation across a config matrix: pick the in-sample-best config up to each fold, apply it to the next. If the picked config drifts fold to fold, that instability is an overfit smell.

walk_forward.py

import edgekit as ek

# sequential blocks: is the edge positive in each period, not just overall?
out = ek.validation.walk_forward(daily_r, k=6, refit=False)
print(out["n_positive"], "of", out["k"], "blocks positive")
for b in out["blocks"]:
    print(b["sharpe"], b["total"], b["positive"])

# expanding re-optimisation: does the picked config stay stable OOS?
wf = ek.validation.walk_forward(M, cfgs=list("abcdefgh"), k=6, refit=True)
print(wf["oos_sharpe"], wf["picks"], wf["stable"])

Scenario. You run refit=True across six folds on a US100 ORB, letting each fold pick the best opening-range length from {15, 30, 45, 60}. Case A: the picks come back [30, 30, 45, 30, 30, 45]— the system keeps choosing ~30–45 minutes, so "a 30-minute range means something" is a real statement and the concatenated OOS Sharpe is trustworthy. Case B: the picks are [15, 60, 30, 60, 15, 45]— every fold crowns a different winner because it is chasing whatever fit that window's noise. Case B can still show a decent average OOS number, but the drifting picks tell you there is no stable parameter underneath — you are watching an overfit happen in slow motion.

✦Config drift is a smell

In refit mode, watch picks. If every fold selects roughly the same configuration, the edge is stable and the parameter means something. If the choice jumps around, you are chasing noise — the walk-forward just made a slow-motion overfit visible.

Purged and embargoed cross-validation#

A subtler leak appears when your labels overlap in time— which they always do in ML meta-labelling, where each trade's outcome resolves over a horizon of future bars. If a training label's outcome window overlaps the test window, information from the test period leaks backward into training. Standard k-fold cross-validation, which ignores time, leaks badly here.

Purging#

Purgefrom the training set any label whose outcome window overlaps the test set's time span. A label opened just before the test block but resolving inside it knows the test period's outcome — drop it.

Embargo#

Even after purging, serial correlation can leak across the train/test boundary. An embargois a gap — at least as long as the label horizon — inserted after the training window so no train label's outcome can resolve inside the test window:

embargo \geq label horizon

This is the firewall. edgekit's ML stack builds it in: walk_forward_windows keeps a train label only if its exit_time falls at or before the embargoed train-window end, and PurgedKFold purges train labels whose horizon overlaps the validation fold plus an embargo tail.

purged.py

import edgekit as ek

# rolling OOS windows with purge + embargo (embargo_hours >= label horizon)
wf = ek.ml.WalkForwardConfig(train_days=7, test_hours=8, embargo_hours=6)
for w in ek.ml.walk_forward_windows(labels, wf):
    tr, te = labels.iloc[w["train_pos"]], labels.iloc[w["test_pos"]]
    # fit on tr, predict te — no train outcome resolves inside the embargo

# purged k-fold for hyperparameter selection inside a training window
for train, val in ek.ml.PurgedKFold(n_splits=4, embargo_frac=0.02).split(labels):
    ...

!The embargo must cover the horizon

If the embargo is shorter than the label horizon, a training label's barrier can still resolve inside the test window — the leak the whole apparatus exists to prevent. Set embargo_hours≥ your label horizon and never shorten it to "get more training data".

What a passing walk-forward looks like#

Signal	Healthy	Overfit
OOS blocks positive	most of k blocks positive	one block carries everything
IS -> OOS Sharpe	modest degradation	collapse (e.g. 1.5 -> 0.1)
Refit config picks	stable across folds	jumps fold to fold
Bad-regime block	small profit or small loss	large loss it never saw in-sample

Some IS→OOS degradation is normal and expected — that is the edge-decay the honest forward projection accounts for. A collapse, or an edge that lives in a single block, is the tell that you fitted noise.

Next: Overfitting detection — parameter plateaus, PBO via CSCV, and the deflated Sharpe that corrects for how many times you tried.