Overfitting detection

The most dangerous backtests are the ones that look best, because the surest way to a great backtest is to try many things and keep the winner. This chapter is about catching that: reading a parameter surface for a plateau vs a knife-edge, estimating the probability of backtest overfitting with CSCV, and deflating the Sharpe by the number of trials you actually ran.

The trap is human, not mathematical: you try things until something works, then remember only the thing that worked. Every discarded variant was a hidden lottery ticket, and the winner is partly just the luckiest ticket. This chapter is three ways of asking "how lucky was my winner, really?" — is one strategy stable when you nudge its dials (plateau vs knife-edge), does being the in-sample best predict anything out-of-sample (PBO), and does the winning Sharpe survive a correction for how many tickets you bought (deflated Sharpe). We will put real-ish numbers through each.

Parameter robustness — plateau vs knife-edge#

A real effect is robust to its knobs. If a strategy works with a 200-bar filter, it should work almost as well at 180 or 240 — the parameter surface is a smooth plateau. An overfit strategy shows a lone towering spike: it is excellent at exactly one value and mediocre on either side, because that one value happened to fit the noise.

param_sweep quantifies this. It runs your strategy across a grid and reports the min, median, and max Sharpe plus the spread. A tight spread with a high median is a plateau; a huge spread dominated by one max is a lucky corner of parameter space.

import edgekit as ek
from edgekit.strategy import ORB

def run(n):
    trades = ek.engine.run_bar_loop(rth, ORB(or_bars=n), warmup=5, bars_per_day=390)
    return ek.trade_stats(trades.r, dates=trades.date)["sharpe"]

sweep = ek.validation.param_sweep(run, grid=range(15, 61, 15))
print(sweep["sharpe_min"], sweep["sharpe_median"], sweep["sharpe_max"], sweep["spread"])
# tight spread, high median -> plateau (trust it); lone max -> overfit

Probability of backtest overfitting (PBO via CSCV)#

When you try many configurations, the question is not "is the winner good in-sample?" but "does being the in-sample winner predict anything out-of-sample?" Combinatorially Symmetric Cross-Validation (Bailey & López de Prado) answers it directly.

How CSCV works#

Take a $T\times N$ matrix of per-period returns — one column per config you tried. Split time into $S$ blocks and form every balanced train/test partition (half the blocks in, half out). In each partition, find the config that was best in-sample, then look up its rank out-of-sample. PBO is the fraction of partitions where the in-sample winner lands in the bottom half out-of-sample:

If picking the in-sample best gives you a coin flip out-of-sample, PBO ≈ 0.5 and your selection has no predictive power. Rule of thumb: $\text{PBO}>0.50$ is overfit; $<0.35$ is acceptable.

import edgekit as ek

# M is a T x Nconfig matrix of per-period returns, one column per config tried
pbo = ek.validation.pbo_cscv(M, S=16)
print(pbo["pbo"], pbo["median_logit"])   # want pbo < 0.35

Scenario. You have a $T\times 24$ matrix of daily returns, one column per ORB filter-combination you tried. CSCV splits time into $S=16$ blocks and forms every balanced half-in/half-out partition. In each one it finds the config that topped the in-sample half, then checks its rank on the held-out half. If pbo comes back 0.55, then more than half the time your in-sample champion landed in the bottom half out-of-sample — picking the backtest winner is worse than a coin flip, so the selection procedure itself has no predictive power and the whole search is rejected regardless of how good the top config looked. If it comes back 0.20, the in-sample winner reliably stays strong out-of-sample and your selection is trustworthy.

The deflated Sharpe ratio#

Try enough strategies and a Sharpe of 2 appears from noise alone. The deflated Sharpe ratio (DSR) corrects the observed Sharpe for the number of trials that produced it, then converts to a probability that the true Sharpe is positive.

The expected maximum Sharpe under the null#

The core insight: if you run $n$ independent strategies with zero true edge, the best of them still has a positive Sharpe by luck, and its expectation grows with $n$. With ${\sigma }_{\text{SR}}$ the standard deviation of Sharpe across trials, the expected maximum is approximately:

where ${\Phi }^{-1}$ is the inverse normal CDF and $\gamma$ the Euler–Mascheroni constant. This $E[\max SR]$ is the benchmark your winner has to beat. The DSR discounts the observed Sharpe by it and applies a skew/kurtosis correction to get a probability:

with ${\hat{\gamma }}_3$ the skew and ${\hat{\gamma }}_4$ the kurtosis of the returns. You want $\text{DSR}>0.95$.

import edgekit as ek

# n_trials = how many configs you actually searched; sr_std = spread of Sharpe across them
dsr = ek.validation.deflated_sharpe(winning_returns, n_trials=100, sr_std=0.03)
print(dsr)   # probability the Sharpe is real; want > 0.95

The three together#

These are complementary views of the same danger. param_sweep asks whether a single strategy is robust to its own knobs; pbo_cscv asks whether your selection procedure across many configs has any OOS power; deflated_sharpe asks whether the winning Sharpe survives the multiple-testing correction. A strategy worth sizing shows a plateau, PBO below 0.35, and DSR above 0.95.

Next: Alpha vs beta — separating genuine skill from leveraged market exposure, and the honest haircut you apply before believing any of it.