edgekit.data

The data / preprocessing plumbing that feeds the causal engine: load bars, resample to swing timeframes, mark DST-correct sessions, split chronologically, fetch from Binance, and cache. It never looks at strategy signals — only at the raw bar frame.

What's inside. The loader (load_bars) and resampler (resample_ohlcv) get raw ticks into the causal OHLC shape. A session toolkit (to_tz, to_ny, minute_of_day, rth_mask) marks regular-trading-hours bars the DST-correct way. integrity_report tells you what the data actually is before you trust it; stitch joins epochs; chronological_split (returning a Split namedtuple) seals a holdout. fetch_binance_klines pulls crypto history over stdlib HTTP, and hashed_parquet_cache memoizes expensive builds. Two rules are enforced throughout: holdout is sealed (read once, at the very end) and sessions are DST-correct (anchored via zoneinfo, never a fixed UTC hour).

→Dependencies

Core paths need only numpy + pandas. Parquet I/O (load_bars on a .parquet path, hashed_parquet_cache) needs the optional pyarrow extra — a missing install raises a pointed ImportError telling you to pip install pyarrow, not an opaque one.

Loading & resampling#

load_bars#

Load bars from a HistoryExporter CSV (time_utc,open,high,low,close,tick_volume) or a processed .parquet split. It validates the schema, sorts the index, and drops duplicate timestamps (keep first) so the frame is monotone and unique — the invariant every indicator and the engine assume.

load_bars(path: str | Path) -> pd.DataFrame

path — a .csv (HistoryExporter format) or .parquet path.

Returns: an OHLCV DataFrame indexed by time_utc with columns open, high, low, close, tick_volume. Raises ValueError on missing columns and a helpful ImportError if a parquet path is read without pyarrow.

load.py

from edgekit import data
bars = data.load_bars("BTCUSDT_5m.csv")
bars.head()      # DatetimeIndex(time_utc) -> open/high/low/close/tick_volume

resample_ohlcv#

Downsample OHLCV bars to a coarser rule — either a TIMEFRAME_RULES key like "H4" or a raw pandas offset alias like "4h". The aggregation is the only correct one: first open, max high, min low, last close, summed volume. Empty daily/weekly weekend buckets are dropped (positive-volume requirement).

resample_ohlcv(df, rule: str, label: str = "left", closed: str = "left") -> pd.DataFrame

Param	Type	Default	Meaning
`df`	pd.DataFrame	—	A finer OHLCV bar frame.
`rule`	str	—	A TIMEFRAME_RULES key ("M15","H4","D1","W1"…) or a raw pandas offset alias.
`label`	str	"left"	Which edge stamps the bucket; "left" = the OPEN timestamp.
`closed`	str	"left"	Which edge is inclusive; "left" includes the left edge.

Returns: the resampled OHLCV DataFrame.

h4 = data.resample_ohlcv(bars, "H4")   # first open, max high, min low, last close, summed vol

!Why label='left' / closed='left' is the causal default

Stamping each bar with the timestamp of its openand including the left edge means a strategy acting at a bar's timestamp only sees bars that have already closed. Flip to right-labelling and a bar carries the timestamp of a candle that hasn't finished forming — silent look-ahead. Keep the defaults unless you know exactly why you are changing them.

Sessions (DST-correct)#

to_tz, to_ny#

Convert a bar index to timezone tz (to_ny is the America/New_York shortcut). tz-naive input is assumed UTC (the OHLC contract) and localized before conversion — the DST-correct way to reason about session boundaries.

to_tz(index, tz: str) -> pd.DatetimeIndex
to_ny(index) -> pd.DatetimeIndex

index — a bar index (tz-naive UTC or already tz-aware).
tz — an IANA timezone name, e.g. "America/New_York", "Europe/London".

Returns: a tz-aware pd.DatetimeIndex in the target zone.

ny = data.to_ny(bars.index)          # US cash-session clock
lon = data.to_tz(bars.index, "Europe/London")

minute_of_day#

Minutes since local midnight (hour*60 + minute) in timezone tz — the primitive behind every session mask. DST-correct because it reads the wall clock in tz, not a fixed UTC offset.

minute_of_day(index, tz: str = "America/New_York") -> np.ndarray

index — a bar index.
tz — the timezone to read the wall clock in (default America/New_York).

Returns: an integer np.ndarray of minutes-since-local-midnight (0–1439).

rth_mask#

Boolean mask selecting regular-trading-hours bars (default US cash 09:30–16:00 NY). Left-inclusive, right-exclusive (start <= t < end), so the 16:00 close bar is excluded. Overnight windows (start > end, e.g. a Sydney session) are handled by OR-ing the wrap-around past midnight.

rth_mask(index, start: str = "09:30", end: str = "16:00",
         tz: str = "America/New_York") -> np.ndarray

Param	Type	Default	Meaning
`index`	DatetimeIndex	—	The bar index to mask.
`start`	str	"09:30"	Session open as "HH:MM" local wall clock.
`end`	str	"16:00"	Session close (exclusive).
`tz`	str	"America/New_York"	Timezone the start/end are expressed in.

Returns: a boolean np.ndarray aligned to index.

mask = data.rth_mask(bars.index)       # 09:30 NY in, 16:00 NY out
session = bars[mask]                    # regular-trading-hours bars only

Inspection & assembly#

integrity_report#

Describe what the data actually is before you trust it: span, gaps, and suspect bars. A gap is a step larger than one bar interval; Friday→Sunday/Monday weekend gaps (normal for FX) are counted separately from intraweek gaps (which indicate missing data).

integrity_report(df: pd.DataFrame, bar_minutes: int) -> dict

df — the bar frame to inspect.
bar_minutes — the nominal bar interval in minutes (5 for M5, 240 for H4), used to size the gap threshold.

Returns a dict with keys: bars, start, end, bar_minutes, span_days, zero_range_bars, weekend_gaps, intraweek_gaps, intraweek_gap_bars_missing, largest_intraweek_gaps (a list of {at, gap}), median_range_pips, spike_bars_range_gt_20x_median, and bad_ohlc_bars.

rep = data.integrity_report(bars, bar_minutes=5)
print(rep["intraweek_gaps"], rep["weekend_gaps"], rep["bad_ohlc_bars"])

stitch#

Concatenate bar frames from multiple epochs (e.g. two export dumps) into one series, deduplicating overlapping timestamps (keep first) and sorting. Returns an empty frame if all inputs are empty or None.

stitch(frames: list[pd.DataFrame]) -> pd.DataFrame

frames — a list of bar frames to combine (empties / None are skipped).

Returns: one sorted, deduplicated DataFrame.

!A stitch does not fill gaps

A real time gap between epochs survives the stitch — the frame is continuous in index only, not in clock time. Do not run one continuous backtest across a stitched seam or the engine will treat the jump as a single (enormous) bar-to-bar move.

Split & chronological_split#

Split a frame into train / validation / holdout by row-count fractions. Splitting by position (not by shuffling) preserves the arrow of time, so a validation bar never precedes a training bar. The result is a Split namedtuple.

class Split(NamedTuple):
    train: pd.DataFrame
    val: pd.DataFrame
    holdout: pd.DataFrame

chronological_split(df, train: float = 0.6, val: float = 0.2) -> Split

Param	Type	Default	Meaning
`df`	pd.DataFrame	—	The frame to split chronologically.
`train`	float	0.6	Fraction of rows in the training slice.
`val`	float	0.2	Fraction in validation; holdout is the remaining 1 − train − val.

Returns: a Split(train, val, holdout) namedtuple of contiguous, non-overlapping slices.

sp = data.chronological_split(bars, train=0.6, val=0.2)
model.fit(sp.train)          # tune on sp.val; NEVER touch sp.holdout until the very end

✕Holdout is sealed

sp.holdout is the last slice — the set you read once, at the very end, to get one honest out-of-sample number. Tuning any parameter against it silently launders the holdout into training data and the “out-of-sample” result becomes a lie. This rule cost the research project real money; edgekit names the slice holdout to make breaking it feel wrong.

Fetching data (open sources)#

Four keyless fetchers pull OHLC history over stdlib HTTP — Binance and Coinbase (crypto), Stooq (equities, indices, FX, commodities EOD), and Yahoo (best-effort everything) — plus a fetch dispatcher that routes to any of them by name. Every one returns the same contract: a frame indexed by time_utc (tz-naive UTC) with float open, high, low, close, volume, sorted and de-duplicated. Parsing is factored into pure private helpers (_parse_stooq_csv, _parse_coinbase_rows, _parse_yahoo_json, _parse_binance_rows) so the wire formats are unit-tested offline; a shared _http_get (SSL-unverified context + retry/backoff) is the only thing that touches the network.

→No API key, ever

All four sources are public, read-only endpoints reached with stdlib urllib only — no requests, no key, no account. Yahoo is best-effort: the chart endpoint is undocumented and can be rate-limited or blocked outright (HTTP 401/429), and its intraday history is range-capped. Prefer Stooq for reliable EOD equity/index/FX data.

fetch#

One entry point over all sources. Routes to the right fetcher by source (one of SOURCES), maps the common intervalto each source's native form where sensible (e.g. "1d" → Stooq "d" / Coinbase 86400 seconds), and — if out is given — also writes a time_utc,open,high,low,close,volume CSV.

fetch(source: str, symbol: str, interval: str = "1d",
      start=None, end=None, out: str | Path | None = None) -> pd.DataFrame

Param	Type	Default	Meaning
`source`	str	—	One of "binance", "coinbase", "stooq", "yahoo".
`symbol`	str	—	The source-native symbol (see each fetcher below).
`interval`	str	"1d"	Common interval ("1d","1w","1mo","1h","5m"…); mapped per source.
`start`	datetime \| str \| int \| None	None	Lower time bound (source-dependent handling).
`end`	datetime \| str \| int \| None	None	Upper time bound; defaults to now where supported.
`out`	str \| Path \| None	None	If given, also writes an OHLCV CSV to this path.

Returns: a frame indexed by time_utc with open, high, low, close, volume (all float). Raises ValueError for an unknown source.

fetch.py

from edgekit import data
spy = data.fetch("stooq", "spy.us", "1d", start="2015-01-01")   # keyless EOD equities
btc = data.fetch("coinbase", "BTC-USD", "1d", start="2020-01-01")
eth = data.fetch("binance", "ETHUSDT", "4h", start="2021-01-01", out="ETHUSDT_4h.csv")
qqq = data.fetch("yahoo", "QQQ", "1d")                          # best-effort

fetch_binance_klines#

Download klines from the Binance public REST API straight into the loader format. It paginates forward from start in 1000-bar pages, retrying on transient errors, until it reaches roughly now; dedups by open time and sorts. Stdlib urllib only — no requests, no API key. This one keeps the HistoryExporter column name tick_volume (holding the Binance base-asset volume) so its CSV round-trips through load_bars.

fetch_binance_klines(symbol: str, interval: str, start,
                     out: str | Path | None = None) -> pd.DataFrame

Param	Type	Default	Meaning
`symbol`	str	—	Binance pair, e.g. "BTCUSDT".
`interval`	str	—	Kline interval — one of 1m,3m,5m,15m,30m,1h,2h,4h,6h,8h,12h,1d,3d,1w.
`start`	datetime \| str \| int	—	Start point: a datetime, a parseable string, or UTC epoch-ms.
`out`	str \| Path \| None	None	If given, also writes a HistoryExporter-style CSV to this path.

Returns: a frame indexed by time_utc with open, high, low, close, tick_volume. Raises ValueError on an unsupported interval.

btc = data.fetch_binance_klines("BTCUSDT", "5m", "2018-01-01", out="BTCUSDT_5m.csv")
h4 = data.resample_ohlcv(btc, "H4")     # ready for a swing strategy

fetch_stooq#

Fetch daily/weekly/monthly OHLC history from Stooq's keyless CSV endpoint — the most reliable open source for equity, index, FX, and commodity end-of-day data.

fetch_stooq(symbol: str, interval: str = "d", start=None, end=None) -> pd.DataFrame

Param	Type	Default	Meaning
`symbol`	str	—	Stooq symbol (see conventions below).
`interval`	str	"d"	Bar size: "d" daily, "w" weekly, "m" monthly.
`start`	date \| str \| None	None	Lower bound; sent as d1=YYYYMMDD.
`end`	date \| str \| None	None	Upper bound; sent as d2=YYYYMMDD.

Symbol conventions. US equities/ETFs are suffixed .us ("aapl.us", "spy.us"); indices are ^-prefixed ("^spx", "^ndq", "^dji"); FX pairs are concatenated codes ("eurusd", "gbpusd"); commodities like "xauusd". A symbol with no volume column (FX) comes back with volume = 0.

spx = data.fetch_stooq("^spx", "d", start="2010-01-01")
eur = data.fetch_stooq("eurusd", "d")   # FX: volume column is 0

fetch_coinbase#

Fetch candles from the Coinbase Exchange public API. granularity is in seconds (Coinbase supports 60 / 300 / 900 / 3600 / 21600 / 86400). The endpoint caps each request at ~300 candles, so when start is given this paginates forward in 300-candle windows to end (default now) and stitches; with no start it returns the single most-recent page.

fetch_coinbase(product: str = "BTC-USD", granularity: int = 86400,
               start=None, end=None) -> pd.DataFrame

Param	Type	Default	Meaning
`product`	str	"BTC-USD"	Coinbase product id, e.g. "BTC-USD", "ETH-USD".
`granularity`	int	86400	Candle size in SECONDS (60/300/900/3600/21600/86400).
`start`	date \| str \| None	None	Lower bound; enables pagination when set.
`end`	date \| str \| None	None	Upper bound; defaults to now.

→Coinbase row order is not OHLC

A Coinbase candle row is [time, low, high, open, close, volume] — low/high before open/close. _parse_coinbase_rows remaps it to the contract for you; the note matters only if you call the API directly.

btc = data.fetch_coinbase("BTC-USD", 86400, start="2020-01-01")   # daily
h1  = data.fetch_coinbase("ETH-USD", 3600, start="2024-01-01")   # hourly

fetch_yahoo#

Fetch OHLC history from Yahoo Finance's chart JSON endpoint. Reads result.timestamp and result.indicators.quote[0]; candles Yahoo pads with nulls (holidays/half-days) are dropped. Best-effort only — undocumented, can be rate-limited or blocked; prefer Stooq for EOD.

fetch_yahoo(symbol: str, interval: str = "1d", start=None, end=None) -> pd.DataFrame

Param	Type	Default	Meaning
`symbol`	str	—	Yahoo ticker: "AAPL", "SPY", "^GSPC", "EURUSD=X", "BTC-USD".
`interval`	str	"1d"	Yahoo interval: "1d","1wk","1mo","1h","5m"…
`start`	datetime \| str \| int \| None	None	period1 (epoch-sec); default full history.
`end`	datetime \| str \| int \| None	None	period2 (epoch-sec); default now.

spy = data.fetch_yahoo("SPY", "1d", start="2015-01-01")   # best-effort; may 401

hashed_parquet_cache#

Memoize an expensive DataFrame build to a content-keyed parquet file. The filename is a hash of key_parts (any JSON-able values — file stamps, config dicts, param tuples); if it already exists it is read, otherwise builder() runs and its result is written. Change any key part and you get a fresh build — no stale-cache bug from mutating inputs under a fixed name.

hashed_parquet_cache(key_parts, builder: Callable[[], pd.DataFrame],
                     cache_dir: str | Path) -> pd.DataFrame

Param	Type	Meaning
`key_parts`	JSON-able	Anything that identifies this build; hashed into the filename.
`builder`	Callable[[], DataFrame]	Zero-arg builder run only on a cache miss.
`cache_dir`	str \| Path	Directory the parquet cache lives in (created if absent).

Returns: the built (or cached) DataFrame. Needs pyarrow.

df = data.hashed_parquet_cache(
    ["BTCUSDT", 5, {"tf": "H4"}],
    lambda: data.resample_ohlcv(data.load_bars("BTCUSDT_5m.csv"), "H4"),
    "cache/",
)

Module constants#

REQUIRED_COLUMNS = ["open","high","low","close","tick_volume"] — the loader schema.
OHLCV_AGG — the resample aggregation {open:first, high:max, low:min, close:last, tick_volume:sum}.
TIMEFRAME_RULES — {"M15":"15min","M30":"30min","H1":"1h","H4":"4h","D1":"1D","W1":"W-FRI"} (weekly anchored on Friday).