The validation gauntlet

Anyone can produce a backtest number. edgekit's entire reason to exist is the disciplined battery that decides whether that number is a real edge or an artefact of luck, look-ahead, or overfitting. This is the crown jewel of the library — run it in order, and stop believing at the first failure.

The three ways alpha is usually fake#

Almost all apparent "alpha" is one of three things wearing a strategy costume. The gauntlet is organised to catch each:

• Beta in disguise. The return is just market exposure. A "trend follower" that is really long the market makes money in a bull run and gives it all back in a bear — that is leverage, not skill. Caught by the is-it-beta regression and the permutation test (which reshuffles away the drift beta rides on).
• Look-ahead. The strategy read information it could not have had. Caught first, by construction and property test (see Causality).
• Overfitting. The pattern was mined from noise — you pulled enough levers that something looked good. Caught by permutation, parameter sweeps, PBO, and the deflated Sharpe.

The nine steps below run in order. Each has one job: try to break the number. If a step fails, you stop — a strategy that dies at step 3 does not get a walk-forward.

Step 1 — Causality / look-ahead audit#

What it checks: that the number belongs to a real, tradeable timeline — every indicator uses prior bars only, and every fill is executable and gap-aware. Why it matters: every later step measures a statistic; if that statistic was produced by reading the future, everything downstream is measuring a ghost. This is why causality is step one.

In edgekit this step is enforced by construction — indicators return unlagged, the caller lags with ek.lag(x, 1), the engine fills gap-aware — and it is guaranteed by the property test that perturbs a future bar and asserts the past does not move. The full treatment is its own page:

import edgekit as ek

upper, lower = ek.indicators.donchian(bars.high, bars.low, 20)
upper = ek.lag(upper, 1)          # a decision at bar i may only see through i-1
# ... entry acts on upper[i]; the engine fills at max(level, open[i]) for longs

Step 2 — Realistic fills and costs#

What it checks: that P&L survives real commission, spread, slippage, and swap. Why it matters: a gross edge that dies once you pay to trade it is not an edge. edgekit charges cost in R (see R-multiples) via CostModel, so a trade's stored r is already net of what it cost to hold:

import edgekit as ek

cost = ek.costs.CostModel(spread_rt=0.0012, swap_day=0.0002)   # 12 bps round trip, 2 bps/day
trades = ek.engine.run_bar_loop(bars, strategy, cost=cost)     # r is net of cost, in R
stats  = ek.trade_stats(trades.r, dates=trades.date)

The bracket engine adds its own realism: a bar that touches both stop and target counts as the stop, and fills happen at the next bar's open. Cost is not a nuisance parameter — it is a test, and step 6 escalates it.

Step 3 — Monte-Carlo permutation test (the decisive one)#

What it checks: whether the edge is distinguishable from what random data would produce. Why it matters: this is the test that separates a real edge from a lucky one, and it is the one edgekit treats as decisive.

The idea: take the market and destroy its structure — the trends and serial correlation a strategy feeds on — while keeping its return distribution exactly. permute_ohlc does this the Masters/NeuroTrader way. It decomposes each log bar into four increments (gap, high−open, low−open, close−open), applies one shared permutation to all four (so each rebuilt bar stays internally consistent and valid), then cumsums back into prices. The set of close-to-close returns is identical to the original; the order — the trend — is gone.

def permute_ohlc(o, h, l, c, rng):
    lo, lh, ll, lc = np.log(o), np.log(h), np.log(l), np.log(c)
    n = len(c)
    pg = lo[1:] - lc[:-1]     # gap: open - prev close
    ph = lh[1:] - lo[1:]      # high - open
    pl = ll[1:] - lo[1:]      # low  - open
    pc = lc[1:] - lo[1:]      # close - open
    j = rng.permutation(n - 1)
    pg, ph, pl, pc = pg[j], ph[j], pl[j], pc[j]   # SAME perm -> shape-consistent bars
    ...                                            # cumsum back into valid OHLC prices

Re-run the strategy on many such synthetic series to build a null distribution of the statistic (total R, profit factor, whatever you chose), then ask how often random data matched or beat the real result. The p-value is:

def mcpt(real_stat, null_fn, n=1000, rng=None):
    """p = (#{null >= real} + 1) / (N + 1).  p < 0.01 = real edge."""
    rng = rng or bootstrap_rng()
    ge = sum(1 for _ in range(n) if null_fn(rng) >= real_stat)
    return (ge + 1) / (n + 1)

The +1 in numerator and denominator is not cosmetic: it counts the observed sample as one of its own permutations, so the p-value can never be exactly 0 — you have not proven impossibility from a finite sample — and the test stays valid. If the real result sits inside the null cloud, it is luck.

import edgekit as ek

real = strategy_total_r(bars)                       # the real statistic

def null_fn(rng):
    po, ph, pl, pc = ek.validation.permute_ohlc(
        bars.open, bars.high, bars.low, bars.close, rng)
    return strategy_total_r(rebuild(po, ph, pl, pc))  # re-run on shuffled bars

p = ek.validation.mcpt(real, null_fn, n=1000)
print(p)     # p < 0.01 -> the edge is real, not luck

Step 4 — Walk-forward across regimes#

What it checks: that the edge persists out-of-sample, sequentially through time — especially in the bad regime (for crypto, the 2022 bear). Why it matters: one favourable sample proves nothing; a real edge shows up in block after block, not just on average.

walk_forward has two modes. The honest sequential-block test (refit=False) splits one strategy's returns into k consecutive blocks and checks the edge is positive in each. The expanding-window re-optimisation (refit=True, with a config matrix and labels) picks the in-sample-best config on data up to each fold and applies it only to the next fold — config drift across folds is an overfit smell.

import edgekit as ek

# sequential blocks: is the edge positive in each period, not just overall?
out = ek.validation.walk_forward(daily_r, k=6, refit=False)
print(out["n_positive"], "of", out["k"], "blocks positive")
for b in out["blocks"]:
    print(b["sharpe"], b["total"], b["positive"])

# expanding re-optimisation: does the picked config stay stable OOS?
wf = ek.validation.walk_forward(M, cfgs=labels, k=6, refit=True)
print(wf["oos_sharpe"], wf["picks"], wf["stable"])

Step 5 — Regime split#

What it checks: where the edge earns — across calendar years, and across market states (trend vs. chop). Why it matters: a strategy positive in only one year, or one that lives entirely in the low-ADX chop regime (or inverts between regimes), is regime-dependent, not robust. A trend system should make its money in trends and bleed little in chop.

import edgekit as ek

# per-year breakdown — recent years are your accidental live-OOS
by_year = ek.validation.regime_by_year(trades.r, trades.date)
print(by_year[["n", "ev", "win", "pf", "total_r"]])

# trend vs chop, split on ADX
adx = ek.indicators.adx(bars.high, bars.low, bars.close, 14)
split = ek.validation.regime_by_adx(bar_net_r, adx, thr=20.0)
print(split["trend_ev"], split["chop_ev"], split["n_trend"], split["n_chop"])

Step 6 — Cost stress ×1/2/3#

What it checks: how the edge behaves as costs escalate. Why it matters: a real edge degrades gracefully at 2× and 3× assumed cost; a fake one collapses. Re-running at inflated cost is also a proxy for a worse execution environment than the backtest assumed. cost_stress re-runs your strategy at each multiplier:

import edgekit as ek

def run(cost):
    trades = ek.engine.run_bar_loop(bars, strategy, cost=cost)
    return ek.trade_stats(trades.r)

grid = ek.validation.cost_stress(run, base=ek.costs.CostModel(), mults=(1.0, 2.0, 3.0))
for mult, stats in grid.items():
    print(mult, "x ->  PF", round(stats["pf"], 2))     # survivor rule: PF > 1 at 2x and 3x

Step 7 — Is it just beta?#

What it checks: whether the return is alpha or leveraged exposure to the benchmark. Why it matters: this is the classic false positive — a trend follower that is really just long the market shows high beta and ~0 alpha. is_it_beta regresses strategy returns on the benchmark: beta is the slope (exposure), alpha the annualised intercept residual (return not explained by riding the benchmark):

import edgekit as ek

out = ek.validation.is_it_beta(strat_daily_ret, buy_and_hold_daily_ret, periods_per_year=365)
print(out["beta"], out["alpha"])
# high beta + ~0 alpha  -> you have leverage, not skill

Step 8 — Parameter robustness (smooth plateau)#

What it checks: whether the edge is robust to its knobs. Why it matters: a real effect is a smooth plateau — e.g. a macro-MA that improves the strategy across 120→300 — while a knife-edge optimum at one lucky value is overfitting. param_sweep reports the min/median/max spread across a grid; a tight spread is a plateau, a huge spread with one towering max is a lucky corner of parameter space:

import edgekit as ek

def run(n):
    trades = ek.engine.run_bar_loop(rth, ek.strategy.ORB(or_bars=n), cost=cost, bars_per_day=390)
    return ek.trade_stats(trades.r, dates=trades.date)["sharpe"]

sweep = ek.validation.param_sweep(run, grid=range(15, 61, 15))
print(sweep["sharpe_min"], sweep["sharpe_median"], sweep["sharpe_max"], sweep["spread"])

Step 9 — Honest forward projection#

What it checks: the number you can actually trade, not the one that looks best. Why it matters: the drawdown-matched backtest is a ceiling, not an expectation. edgekit never hides the haircut. Two multipliers, applied to the sized projection:

• Edge decay ≈ ×0.85 if the edge is permutation-validated (worse if it is not). The future regime is assumed less favourable than the past.
• Size-down ≈ ×0.75 because live drawdowns run deeper than the single lucky historical one. Size against the block-bootstrap dd95 (a bad-but-not-tail drawdown), not the realised max.

import edgekit as ek

# size against a realistic worst case, not the one lucky historical drawdown
dd = ek.validation.dd95(daily_r, block=5, horizon=252, n=15000)   # 95th-pct max DD
sized = ek.sizing.size_to_dd(daily_r, dd_budget=0.095, account=100_000, daily_cap=0.045)

backtest_annual = sized["dollar_per_r"] * daily_r.sum() / len(...)  # illustrative
honest = backtest_annual * 0.85 * 0.75         # edge-decay x size-down -> plan around this

The overfit battery#

Alongside the nine steps, two diagnostics from López de Prado guard against the subtler failure of having tried too many configs:

• pbo_cscv — Probability of Backtest Overfitting via CSCV. Feed a T × Nconfig matrix of per-period returns (one column per config you tried); over every balanced train/test split it records the OOS rank of the in-sample-best config. PBO > 50% means your selection has no OOS predictive power; < 35% is acceptable.
• deflated_sharpe — try enough knobs and a Sharpe of 2 appears from noise alone. DSR discounts the observed Sharpe by the expected maximum Sharpe under the null across n_trials, with a skew/kurtosis correction. Want DSR > 0.95.

import edgekit as ek

pbo = ek.validation.pbo_cscv(M, S=16)              # M = T x Nconfig return matrix
print(pbo["pbo"], pbo["median_logit"])             # want pbo < 0.35

dsr = ek.validation.deflated_sharpe(winner_returns, n_trials=40, sr_std=0.03)
print(dsr)                                          # want > 0.95

A worked rejection — the ORB#

The reference recipe (examples/orb_gauntlet.py) runs a bare 30-minute opening-range breakout on US100 M1 through the whole battery. It is a famous, intuitive strategy that looks tradeable — and the gauntlet kills it, which is exactly what the gauntlet is for:

Here is the honest nuance that makes the ORB such a good teaching case: its permutation p is low, so the timing is not pure noise — a real breakout does cluster its entries better than a coin flip. And yet the profit factor never clears 1, not even at nominal cost, so the strategy loses money. The permutation test asks "is the timing structure real?"; it does not ask "does this beat the spread?". The ORB clears the first bar and fails the second. Beating a coin-flip is not an edge — and telling the two apart is the whole job of the gauntlet.

See also#

• edgekit.validation — every function in the gauntlet, module by module.
• Causality — step 1, the correctness foundation everything else rests on.
• R-multiples — the unit the whole gauntlet operates in.
• edgekit.costs — the cost model and the cost-stress harness (steps 2 and 6).