edgekit.validation

The validation gauntlet — the crown jewel. Assume every edge is fake until proven otherwise; this module is the machinery that tries to break your result before the market does. numpy + pandas only (the permutation and normal-quantile math is implemented here so import edgekit never needs scipy).

What's inside. Two null generators (permute_ohlc, permute_returns) feed the Monte-Carlo permutation test (mcpt) — the decisive step. Around it sit the rest of the gauntlet: overfitting probability (pbo_cscv), deflated Sharpe (deflated_sharpe), honest walk-forward (walk_forward), IS/OOS cuts (oos_split, is_oos_split), the alpha-vs-beta regression (is_it_beta), a plateau detector (param_sweep), regime splits (regime_by_year, regime_by_adx), and the block-bootstrap drawdown distribution (block_bootstrap_mc, dd95). cost_stress is re-exported here from edgekit.costs as gauntlet step 6.

__all__ = ["permute_ohlc", "permute_returns", "mcpt", "pbo_cscv", "deflated_sharpe",
           "walk_forward", "oos_split", "is_oos_split", "is_it_beta", "param_sweep",
           "regime_by_year", "regime_by_adx", "block_bootstrap_mc", "dd95", "cost_stress"]

The nulls#

permute_ohlc#

The Masters / NeuroTrader OHLC bar-permutation — the correct null for trend and breakout systems. It decomposes each log bar into four increments (gap, high−open, low−open, close−open), applies one shared permutation to all four so rebuilt bars stay internally valid (high ≥ open/close/low), then cumsums back into prices. This destroys serial structure — trends, autocorrelation — while preserving the return marginal and the bar shapes. Bar 0 is the untouched anchor.

permute_ohlc(o, h, l, c, rng) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]

• o, h, l, c — open/high/low/close arrays (price levels).
• rng — a numpy Generator (use edgekit.core.bootstrap_rng() for reproducibility).

Returns a (open, high, low, close) tuple of arrays — a fresh, structure-free but marginal-preserving market.

permute_returns#

Shuffle a return series — the null for anything that acts on returns directly. Breaks time-ordering while keeping the exact return marginal (it is a reshuffle, not a re-draw).

permute_returns(ret, rng) -> np.ndarray

• ret — a 1-D return series.
• rng — numpy Generator.

The decisive test#

mcpt#

Monte-Carlo permutation p-value. You supply your real statistic and a null_fn(rng) that produces one statistic on freshly-permuted data; mcpt calls it n times and counts how often the null matches or beats real.

mcpt(real_stat: float, null_fn, n: int = 1000, rng=None) -> float

Returns a single float p-value in (0, 1].

import numpy as np, pandas as pd
from edgekit import validation as v
from edgekit.core import bootstrap_rng

def donchian_stat(o, h, l, c, n=20):
    """Total log-return captured while long a 20-bar Donchian breakout."""
    c = np.asarray(c, float)
    up = pd.Series(h).rolling(n).max().shift(1).to_numpy()
    dn = pd.Series(l).rolling(n).min().shift(1).to_numpy()
    ret = np.diff(np.log(c), prepend=np.log(c[0]))
    pos, total = 0, 0.0
    for i in range(n + 1, len(c)):
        if pos == 0 and c[i - 1] > up[i - 1]:   pos = 1
        elif pos == 1 and c[i - 1] < dn[i - 1]: pos = 0
        total += pos * ret[i]
    return total

real = donchian_stat(o, h, l, c)

# null_fn re-runs the SAME statistic on a permuted market
def null_fn(rng):
    return donchian_stat(*v.permute_ohlc(o, h, l, c, rng))

p = v.mcpt(real, null_fn, n=1000, rng=bootstrap_rng())
print(f"permutation p = {p:.4f}")     # p < 0.01 -> real edge

Overfitting & significance#

pbo_cscv#

Probability of Backtest Overfitting via CSCV (Bailey & Lopez de Prado). Given a matrix of per-period returns with one column per config you tried, it splits time into S blocks, and over every balanced train/test partition picks the in-sample-best config and records where it lands out-of-sample. PBO is the fraction of splits where the IS-winner lands in the bottom half OOS.

pbo_cscv(M, S: int = 16) -> dict

• M — a T × Nconfig matrix of per-period returns (one column per config).
• S — number of contiguous time blocks to partition (default 16).

Returns a dict: pbo (fraction), median_logit. Rule of thumb: > 0.50 = overfit; < 0.35 = acceptable.

deflated_sharpe#

The Deflated Sharpe Ratio: the probability the observed Sharpe is real given you ran n_trials configurations. It discounts the Sharpe by the expected maximum Sharpe under the null across those trials, then converts to a probability with a skew/kurtosis correction.

deflated_sharpe(returns, n_trials: int = 1, sr_std: float | None = None,
                periods_per_year: float = 252.0) -> float

Returns a probability float in [0, 1]. Want DSR > 0.95.

dsr = v.deflated_sharpe(winning_returns, n_trials=100, sr_std=0.03)

Out-of-sample & walk-forward#

walk_forward#

Two modes, selected by refit. The honest default splits one strategy's returns into k consecutive blocks and checks each is positive (an edge should be spread across time, not concentrated in one block). The refit mode does expanding-window re-optimisation across a config matrix.

walk_forward(M, cfgs=None, k: int = 6, refit: bool = False,
             periods_per_year: float = 252.0) -> dict

• M — a 1-D single-strategy return series (refit=False) or a T × Nconfig matrix (refit=True).
• cfgs — config labels aligned to the matrix columns (only for refit=True).
• k — number of sequential blocks (default 6).
• refit — False: sequential-block test; True: expanding-window re-optimisation.

refit=False returns refit, k, blocks (list of {n, sharpe, total, positive}), n_positive. refit=True returns refit, oos_sharpe, picks, n_folds, oos_returns, stable.

out = v.walk_forward(daily_r, k=6, refit=False)
print(out["n_positive"], "/", out["k"])   # want most blocks positive

# expanding-window re-optimisation across a config matrix
out = v.walk_forward(M, cfgs=list("abcdefgh"), k=6, refit=True)
print(out["oos_sharpe"], out["picks"])

oos_split#

Hard calendar cut into in-sample / out-of-sample date-indexed slices — used to measure IS→OOS Sharpe degradation.

oos_split(df, cut: str = "2023-01-01")   # -> (in_sample, out_of_sample)

is_oos_split#

Positional cut on a bare array: chronological split at fraction frac (no shuffle, preserving time's arrow).

is_oos_split(M, frac: float = 0.55)   # -> (IS, OOS) arrays

a, b = v.is_oos_split(np.arange(100), frac=0.55)   # len(a)==55, len(b)==45

Alpha vs beta#

is_it_beta#

Regress strategy returns on a benchmark: is this alpha, or just the market in a strategy costume? A trend-follower that is really just long the market shows high beta and ~0 alpha.

is_it_beta(strat_ret, bench_ret, periods_per_year: float = 252.0) -> dict

• strat_ret — the strategy return series.
• bench_ret — the benchmark (e.g. buy-and-hold) return series, aligned.
• periods_per_year — annualisation for the intercept (default 252).

Returns a dict: beta (the slope), alpha (the annualised intercept residual).

out = v.is_it_beta(strat_daily, btc_buyhold_daily)
print(out["beta"], out["alpha"])   # want beta near 0 and alpha > 0

Robustness & regime#

param_sweep#

Plateau detector: a real edge is robust to its knobs, an overfit one is a spike. run_fn(params) returns a Sharpe (or a dict containing 'sharpe') for each point in the grid.

param_sweep(run_fn, grid) -> dict

Returns: sharpe_min, sharpe_median, sharpe_max, spread, n. A tight spread across the grid is a plateau (good); a lone spike is overfit.

out = v.param_sweep(lambda p: run_config(entry_n=p), grid=range(15, 30))
print(out["sharpe_median"], out["spread"])

regime_by_year#

Per-year breakdown of an R-stream — is the edge spread across years, or one lucky one?

regime_by_year(r, dates) -> pd.DataFrame

Returns a DataFrame indexed by calendar year, columns n, ev, win, pf, total_r.

df = v.regime_by_year(trades.r, trades.date)
print(df)   # year-by-year n / ev / win / pf / total_r

regime_by_adx#

Split net returns into chop (ADX < thr) vs trend (ADX >= thr). Trend systems should make money in trends and bleed little in chop.

regime_by_adx(net, adx, thr: float = 20.0) -> dict

Returns: chop_ev, trend_ev, n_chop, n_trend.

Forward risk#

block_bootstrap_mc#

Stationary block-bootstrap of a daily-return stream into forward wealth & drawdown distributions. Resampling 5-day blocks (rather than i.i.d. days) preserves autocorrelation and vol-clustering, so the drawdown distribution is honest rather than optimistic.

block_bootstrap_mc(returns, block: int = 5, horizon: int = 252,
                   n: int = 20000, rng=None) -> dict

Returns: terminal (array of terminal cum-returns), drawdowns (array of max DDs). Raises ValueError if fewer than block+1 returns.

dd95#

The 95th-percentile max drawdown from the block-bootstrap — the number to size against, more conservative than the single realised historical drawdown.

dd95(returns, block: int = 5, horizon: int = 252, n: int = 15000, rng=None) -> float

worst = v.dd95(daily_r, horizon=252)   # size the book against THIS, not the lucky historical DD

cost_stress#

Re-exported from edgekit.costs as gauntlet step 6. Re-run a strategy at escalating cost and return {mult: metrics}; a real edge degrades gracefully, a fake one collapses. Survivor rule of thumb: PF must stay > 1 at 2x and 3x.

cost_stress(run_fn, base=None, mults=(1.0, 2.0, 3.0)) -> dict

from edgekit.strategy import ORB
from edgekit import trade_stats
def run(cost): return trade_stats(ORB(or_bars=30, target_r=2.0).backtest(rth, cost=cost, warmup=5, bars_per_day=390).r)
grid = v.cost_stress(run)   # {1.0: {...}, 2.0: {...}, 3.0: {...}}

See also#

• The validation gauntlet — the concept, end to end.
• edgekit.strategy — the R-multiple backtests you feed into these tests.
• edgekit.costs — the source of cost_stress.