Time-series statistics

Everything in classical statistics quietly assumed independent observations. Financial data laugh at that: returns cluster, prices trend, volatility begets volatility. This chapter rebuilds the essentials for dependent data — stationarity, autocovariance, white noise versus iid — works the AR(1) model completely (mean, variance, ACF, the unit root at $\phi =1$), derives how autocorrelation shrinks your effective sample size, and demonstrates the spurious-regression trap that has fooled a century of researchers.

Stationarity: strict and weak#

A process $\{X_t\}$ is strictly stationary if every finite-dimensional distribution is shift-invariant: $(X_{t_1},\dots ,X_{t_k})\stackrel{d}{=}(X_{t_1+h},\dots ,X_{t_k+h})$ for all $k,h$ and time points. It is weakly (covariance) stationary if only the first two moments are shift-invariant:

Neither implies the other in general. Strict + finite second moments $\Rightarrow$ weak; but an iid sequence of Cauchy variables is strictly stationary and not weakly stationary (no variance exists — see the distribution zoo). Conversely a weakly stationary process can have wildly time-varying higher moments. Counterexamples to hold in mind:

• White noise: stationary (both senses if iid Gaussian). The boring baseline.
• Random walk $P_t=P_{t-1}+{\epsilon }_t$: not stationary in any sense — $\mathrm{Var}(P_t)=t{\sigma }^2$ grows without bound. Its increments are stationary.
• Deterministic seasonality $X_t=\cos (2\pi t/12)+{\epsilon }_t$: the mean depends on $t$ — not stationary, though it looks tame.
• GARCH returns: weakly stationary (constant unconditional moments) yet the conditional variance moves every day — stationarity is about unconditional distributions.

Autocovariance, autocorrelation, and white noise vs iid#

For a weakly stationary process, define the autocovariance $\gamma (h)=\mathrm{Cov}(X_t,X_{t+h})$ and autocorrelation $\rho (h)=\gamma (h)/\gamma (0)$. Basic properties: $\gamma (h)=\gamma (-h)$ (symmetry), $|\gamma (h)|\le \gamma (0)$ (Cauchy–Schwarz, proved in Inequalities & tail bounds), and the whole function must be positive semi-definite — because $\mathrm{Var}(\sum a_i{X}_{t_i})\ge 0$ for any weights, exactly the argument that makes covariance matrices PSD in Joint distributions. Not every damped sequence is a valid ACF.

White noise $\{{\epsilon }_t\}\sim \mathrm{WN}(0,{\sigma }^2)$ means mean zero, constant variance, and $\gamma (h)=0$ for $h\ne 0$ — uncorrelated, nothing more. IID is strictly stronger: independence kills all functional relationships, not just linear ones. The canonical finance example of the gap: GARCH returns are white noise (uncorrelated — no linear predictability, consistent with a martingale null) while their squares are strongly autocorrelated — volatility is predictable even when direction is not. This ARCH effect is the single most robust stylised fact of asset returns, and the reason ARIMA & GARCH is a chapter and not a footnote.

AR(1), worked completely#

The workhorse model of mean reversion. With $\{{\epsilon }_t\}\sim \mathrm{WN}(0,{\sigma }^2)$:

Worked numbers. Daily spread with $c=0.5,\phi =0.9,\sigma =1$. Long-run mean $\mu =0.5/0.1=5$; stationary variance $1/(1-0.81)=5.26$, sd $2.29$ — more than double the shock sd, because shocks accumulate before decaying. Autocorrelations: $\rho (1)=0.9,\rho (5)=0.59,\rho (20)=0.12$; half-life $\ln 2/\ln (1/0.9)\approx 6.6$ days — the timescale a mean-reversion trade on this spread should hold for.

The unit root. At $\phi =1$ (and $c=0$) the recursion is the random walk: $X_t=X_0+{\sum }_{s\le t}{\epsilon }_s$, with $\mathrm{Var}(X_t)=t{\sigma }^2$ and $\mathrm{Corr}(X_t,X_{t+h})=\sqrt{t/(t+h)}\to 1$: shocks are permanent, nothing mean-reverts, and every formula above (mean $c/(1-\phi )$, variance ${\sigma }^2/(1-{\phi }^2)$) divides by zero. The boundary between $\phi =0.99$ (stationary, slow) and $\phi =1$ (nonstationary) is statistically delicate but conceptually absolute — it is the difference between a spread you can trade back to fair value and one that has no fair value.

MA(1) and the AR/MA signatures#

The moving-average counterpart: $X_t=\mu +{\epsilon }_t+\theta {\epsilon }_{t-1}$. Mean $\mu$; variance $\gamma (0)=(1+{\theta }^2){\sigma }^2$; first autocovariance $\gamma (1)=\theta {\sigma }^2$ (only the shared shock ${\epsilon }_{t-1}$ overlaps); and $\gamma (h)=0$ for $h\ge 2$ — the ACF cuts off dead after lag 1:

That bound is a classic result in itself: no MA(1) can have first-lag autocorrelation above one half (maximise $\theta /(1+{\theta }^2)$ by calculus, or note $(1-\theta {)}^2\ge 0$). The identification signatures, read off any pair of ACF/PACF plots:

Autocorrelation and the sample mean: effective sample size#

Now the practical payoff. For a stationary series, the variance of the sample mean is not $\gamma (0)/n$:

Spurious regression#

The most expensive mistake in applied time series: regress one trending series on another, unrelated one, and the regression looks superb. Granger and Newbold’s classic setup: two independent random walks $x_t,y_t$, regression $y_t=\alpha +\beta x_t+u_t$. The truth is $\beta =0$, yet the t-test rejects wildly. Why: the OLS t-statistic assumes iid errors, but here the residual inherits the random walks’ nonstationarity — $u_t$ is itself (near) a random walk, its autocorrelation approaches 1, the effective sample size approaches a handful, and both series wander persistently so any finite window shows some incidental co-drift that OLS eagerly fits. Asymptotically the t-statistic diverges like $\sqrt{n}$ and $R^2$ converges not to zero but to a nondegenerate random variable — more data makes the illusion stronger.

import numpy as np

rng = np.random.default_rng(11)
n, trials = 250, 2000                    # one 'year' of daily levels, many experiments
rej, r2s = 0, []

for _ in range(trials):
    x = np.cumsum(rng.normal(size=n))    # independent random walks
    y = np.cumsum(rng.normal(size=n))
    X = np.column_stack([np.ones(n), x])
    b = np.linalg.lstsq(X, y, rcond=None)[0]
    u = y - X @ b
    s2 = u @ u / (n - 2)
    se_b = np.sqrt(s2 * np.linalg.inv(X.T @ X)[1, 1])
    t = b[1] / se_b
    rej += abs(t) > 1.96
    r2s.append(1 - (u @ u) / ((y - y.mean()) @ (y - y.mean())))

print(f"share with |t| > 1.96 : {rej/trials:.2f}   (should be 0.05; is ~0.8)")
print(f"median R^2            : {np.median(r2s):.2f}   (truth: zero relationship)")
# repeat with returns np.diff(...) instead of levels: rejection drops to ~5%.

The repair kit: difference to returns before regressing (unit root removed, classical inference restored), or — when a genuine long-run level relationship is the hypothesis — test for cointegration (the residual of the levels regression must itself be stationary), which is the statistical foundation of pairs trading. The Dickey–Fuller idea supplies the test: fit $\Delta X_t=\alpha +\beta X_{t-1}+{\epsilon }_t$ and test $\beta =0$ (unit root) against $\beta <0$ (mean reversion — a negative pull toward the mean whenever the level is high). The subtlety earning its own tables: under the unit-root null the t-statistic does notfollow a t distribution (the regressor is nonstationary, so CLT machinery fails); its Dickey–Fuller distribution has a 5% critical value near $-2.86$, well beyond the usual $-1.65$. Ignoring this rejects stationarity far too often — the same trap in reverse.

Finally, the diagnostic that wraps up the chapter’s toolkit: the Ljung–Box test aggregates the first $h$ sample autocorrelations into

(each ${\hat{\rho }}_k\approx \mathcal{N}(0,1/n)$ under the null, so the sum of squares is approximately chi-square — the small-sample factors sharpen the approximation). Run it on returns to test for linear predictability, and on squaredreturns to test for ARCH effects; a strategy’s residual P&L failing Ljung–Box is either exploitable structure you missed or dependence your standard errors ignored.

Practice problems#

Next: with dependence understood, the last tool is brute force done right — generating random draws, variance reduction, and the $1/\sqrt{n}$ law of simulation. Simulation & Monte Carlo.