Proving an edge

A backtest is a hypothesis, not a result. This guide runs the ORB from the previous page through the full validation gauntlet — permutation test, cost-stress, is-it-beta, walk-forward, param-sweep, PBO and deflated Sharpe — and shows what each step is trying to kill. The ORB is the honest case study: a plausible, famous strategy that the gauntlet rejects.

→Prime directive

Assume the edge is fake. Your job is to disprove it. What survives every test below is the part you get to trade — and most candidates survive nothing. If any one test fails decisively, you stop. You do not keep torturing the data until it confesses.

All of this assumes you already have the trade frame and stats from the first-backtest guide:

prove.py

import numpy as np
import pandas as pd
import edgekit as ek
from edgekit.core import bootstrap_rng

bars = ek.data.load_bars("US100_M1.csv")
rth  = bars[ek.data.rth_mask(bars.index, start="09:30", end="16:00", tz="America/New_York")]
orb  = ek.strategy.ORB(or_bars=30, target_r=2.0)
trades = orb.backtest(rth, warmup=5, bars_per_day=390)
stats  = ek.trade_stats(trades.r.to_numpy(), dates=trades.date)
real   = float(trades.r.sum())   # -570.6R — the statistic to beat

Step 1 — the permutation test#

The mcpt Monte-Carlo permutation test asks the only question that matters about timing: how often does random data reproduce this result? You write a null_stat(rng) that permutes the bars and re-runs the exact same strategy, then mcpt calls it many times and counts how often the shuffled result matches or beats the real one.

For a breakout system the right null is permute_ohlc — the Masters/NeuroTrader bar permutation. It decomposes each log bar into four increments, applies one shared shuffle so the rebuilt bars stay internally valid, and cumsums back to prices. This destroys serial structure (trends, autocorrelation, intraday momentum) while preserving the return marginal and bar shapes. A breakout run on that data should make nothing.

o, h, l, c = (rth[x].to_numpy(float) for x in ("open", "high", "low", "close"))
idx = rth.index

def null_stat(rng: np.random.Generator) -> float:
    po, ph, pl, pc = ek.validation.permute_ohlc(o, h, l, c, rng)
    shuffled = pd.DataFrame({"open": po, "high": ph, "low": pl, "close": pc}, index=idx)
    t = orb.backtest(shuffled, warmup=5, bars_per_day=390)   # SAME strategy, permuted world
    return float(t.r.sum()) if len(t) else 0.0

p = ek.validation.mcpt(real, null_stat, n=100, rng=bootstrap_rng())
print(f"permutation test: p = {p:.4f}")
# permutation test: p = 0.0396  ->  the ORB's timing beats the shuffle

The p-value is (#{null >= real} + 1) / (N + 1) — the +1 guarantees p > 0. Reading the scale:

p-value	Verdict on timing
`p < 0.01`	the timing is essentially never reproduced by random data
`0.01 – 0.05`	marginal — the timing is suggestive, not conclusive
`p > 0.05`	no signal — random entries do just as well (p ~ Uniform(0,1))

Here is the trap the ORB is built to teach. p = 0.04 says its entry timing genuinely beats random-entry shuffles: breaking a real opening range picks better moments than firing at random bars. And yet — as the next step shows — the strategy still loses money net of costs. The permutation test clears the coin-flip; it says nothing about whether you clear the spread.

!The permutation test is necessary, not sufficient

A low permutation p is a gate you must pass, not a licence to trade. It only asks whether your signal beats random timing on the same bars — it does not know your costs, and it does not know that a negative total R can still be less-negative than the shuffles. Beating a coin-flip is not enough. You must beat the spread, and that is a different test.

Step 2 — cost-stress (x1 / x2 / x3) — the step that kills it#

Many "edges" are just an underestimate of transaction cost — and some, like the ORB, do not even have a gross edge left over once honest friction is charged. cost_stress (re-exported from costs) re-runs the strategy at escalating cost and returns {mult: metrics}. The survivor rule of thumb: profit factor must stay above 1 at 2x and 3x. The ORB is already below 1 at 1x.

from edgekit.costs import cost_stress

def run(cost):
    return ek.trade_stats(orb.backtest(rth, cost=cost, warmup=5, bars_per_day=390).r.to_numpy())

grid = cost_stress(run)                    # {1.0: {...}, 2.0: {...}, 3.0: {...}}
for mult, s in grid.items():
    print(f"  {mult:.0f}x -> PF {s['pf']:.2f}  EV {s['ev_r']:+.3f}R")
#   1x -> PF 0.71  EV -0.214R
#   2x -> PF 0.45  EV -0.462R
#   3x -> PF 0.29  EV -0.690R

PF 0.71 → 0.45 → 0.29 is a strategy with no positive expectancy to defend: it loses at real cost and loses more at double and triple. This is the decisive failure. A real edge degrades gracefully and stays above PF 1; the ORB never gets there. The gauntlet is a chain, so in practice you stop here — the remaining steps below are shown to complete the tour, but the verdict is already in.

Step 3 — is it beta?#

A directional strategy that is secretly just long the market will look great in a bull run and evaporate in a chop. is_it_betaregresses the strategy's daily returns on a benchmark (here buy-and-hold US100) and splits the result into alpha (annualised intercept) and beta (slope). You want meaningful positive alpha and low beta.

daily_r = trades.set_index("date").r.groupby(lambda t: t.normalize()).sum()
bench   = rth["close"].pct_change().resample("1D").sum()   # US100 daily return
aligned = pd.concat([daily_r, bench], axis=1).fillna(0.0)

res = ek.validation.is_it_beta(aligned.iloc[:, 0].to_numpy(), aligned.iloc[:, 1].to_numpy())
print(f"beta {res['beta']:+.2f}  alpha {res['alpha']:+.1%}/yr")
# the ORB is roughly beta-neutral, but with NEGATIVE alpha - there is no return to attribute

Step 4 — walk-forward#

Is a result spread across the whole history, or one lucky stretch? walk_forward with refit=False is the honest sequential-block test: split the daily-R series into k consecutive blocks and check each.

out = ek.validation.walk_forward(daily_r.to_numpy(), k=6, refit=False)
print(f"{out['n_positive']} / {out['k']} blocks positive")
for b in out["blocks"]:
    print(f"  n={b['n']:4d}  sharpe {b['sharpe']:+.2f}  total {b['total']:+.1f}  positive={b['positive']}")

For the ORB the blocks are mostly negative — the bleed is persistent, not a single bad regime, which is exactly what you expect from a no-edge strategy paying costs every day. (For a genuine edge you want six-out-of-six positive; five-of-six with one small negative block is normal.) Complement this with ek.validation.regime_by_year(trades.r, trades.date) for the calendar-year breakdown.

Step 5 — param-sweep (the plateau test)#

A real edge is a broad plateau in parameter space; an overfit one is a lonely spike; a no-edge strategy is a plateau of uniformly bad numbers. param_sweep runs run_fn(params) -> sharpe over a grid and reports the spread of Sharpes.

def run_cfg(prm):
    t = ek.strategy.ORB(or_bars=prm["or_bars"], target_r=prm["target_r"]).backtest(
        rth, warmup=5, bars_per_day=390)
    return ek.trade_stats(t.r.to_numpy(), dates=t.date)["sharpe"]

grid = [{"or_bars": ob, "target_r": tr} for ob in (15, 30, 45, 60) for tr in (1.5, 2.0, 2.5)]
sweep = ek.validation.param_sweep(run_cfg, grid)
print(sweep)   # {'sharpe_min':..., 'sharpe_median':..., 'sharpe_max':..., 'spread':..., 'n':12}

Here sharpe_median is negative and the whole grid is a flat, losing plateau — there is no magic opening-range length or target that rescues it. Consistency around a bad number is still bad.

Step 6 — PBO and deflated Sharpe#

These two quantify the overfitting risk of having tried many things, and normally you only reach them for a candidate that already cleared costs. pbo_cscv (Probability of Backtest Overfitting, via CSCV) takes a T × Nconfig matrix — one return column per config you tried — and estimates how often the in-sample winner lands in the bottom half out-of-sample. Under 35% is acceptable; over 50% means your selection process is overfit.

# build a returns matrix, one column per config from your sweep
cols = {}
for ob in (15, 30, 45, 60):
    for tr in (1.5, 2.0, 2.5):
        t = ek.strategy.ORB(or_bars=ob, target_r=tr).backtest(rth, warmup=5, bars_per_day=390)
        cols[f"{ob}_{tr}"] = t.set_index("date").r.groupby(lambda d: d.normalize()).sum()

M = pd.DataFrame(cols).fillna(0.0)
pbo = ek.validation.pbo_cscv(M.to_numpy(), S=16)
print(f"PBO {pbo['pbo']:.0%}  median_logit {pbo['median_logit']:+.2f}")   # < 35% good

deflated_sharpe then discounts the observed Sharpe by the maximum you would expect to see by chance across n_trialsconfigs, and returns the probability the Sharpe is real. Feed it the number of configs you actually tried; you want DSR > 0.95 (a negative Sharpe never gets there).

dsr = ek.validation.deflated_sharpe(daily_r.to_numpy(), n_trials=len(cols), sr_std=0.03)
print(f"deflated Sharpe probability: {dsr:.2f}")   # far below 0.95 for the ORB

!Stop believing at the first failure

The gauntlet is a chain, not a menu. Once cost-stress showed PF < 1, the ORB was dead — no amount of pretty walk-forward or param-plateau rescues it. And the reverse trap is just as fatal: do not re-run with different parameters until one test passes and then report that one. That is the overfitting the whole battery exists to catch. A strategy is only as strong as its weakest gauntlet step.

The verdict on the ORB#

Permutation p ≈ 0.04 — the timing beats a coin-flip, but that is not an edge.
Cost-stress PF 0.71 / 0.45 / 0.29 — net-negative at 1x and worse under stress. This is the rejection.
Beta-neutral but with negative alpha — there is no return to attribute.
Persistently negative walk-forward blocks and a uniformly losing param plateau — not one bad regime.

That is what the gauntlet doing its job looks like. "Prove the edge" cuts both ways: a famous, intuitive strategy that looks tradeable is killed the moment costs are priced in. Most candidates should die here — and the examples exist to show the library, not to hand out alpha.

Next#

The validation gauntlet — the concept and the full checklist.
API · validation — every function used above.
Sizing for a prop-firm challenge — the mechanics, for an edge that does survive.
Example: the ORB gauntlet — the whole thing as one runnable script.