Conditional expectation

Part I treated $E [X ∣ Y]$ as a passing tool; this chapter makes it the main character. The idea unfolds as a ladder — condition on an event, then on a value, then realise the whole family of values assembles into a random variable — and once you see $E [X ∣ Y]$ as an object in its own right, three of the most useful theorems in probability fall out almost for free: the tower property, the law of total variance, and the fact that $E [X ∣ Y]$ is the best possible predictor of $X$ from $Y$ . The same machinery, pointed at a process instead of a pair, becomes first-step analysis — the one-line solver for “how long until…” questions — and Wald’s identity for random sums.

✦Intuition — a best guess that is itself random

Before observing anything, your best guess for $X$ is the single number $E [X]$ . After observing $Y$ , your best guess depends on what you saw: it is a rule, “if $Y = y$ , guess $m (y)$ .” Feed the random $Y$ into that rule and the guess itself becomes random: $E [X ∣ Y] = m (Y)$ . Everything in this chapter is a consequence of taking that seriously. Average the guess over what you might see and you recover the unconditional answer (tower property). Split $X$ ’s uncertainty into “scatter around the guess” plus “scatter ofthe guess” and you get the law of total variance. Ask which function of $Y$ predicts $X$ with least squared error and the answer is — this same object. One definition, three theorems, and a technique (condition on the first step) that dissolves whole families of problems.

The ladder: event, value, random variable#

Rung 1 — conditioning on an event. For an event $A$ with $P (A) > 0$ , the conditional expectation of a discrete $X$ is the ordinary expectation under the renormalised measure of the axioms chapter:

E [X ∣ A] = x \sum x P (X = x ∣ A) = \frac{E [ X 1 _{A} ]}{P ( A )} .

A number. For a fair die, $E [X ∣ X even] = \frac{2 + 4 + 6}{3} = 4$ .

Rung 2 — conditioning on a value. Take $A = {Y = y}$ and let $y$ range. Each value of $y$ produces a number, so we have built an ordinary function,

m (y) = E [X ∣ Y = y] = x \sum x P (X = x ∣ Y = y),

the regression function of $X$ on $Y$ — the same object estimated from data in regression & inference. (In the continuous case the sum becomes an integral against the conditional density $f_{X ∣ Y} (x ∣ y)$ from joint distributions; nothing conceptual changes.)

Rung 3 — conditioning on the random variable. Now compose the function with the random input:

E [X ∣ Y] := m (Y) .

This is the step that upgrades conditioning from a computation to a concept. $E [X ∣ Y]$ is a random variable — a function of $Y$ and of nothing else. Before $Y$ is revealed you do not know which best guess you will make, but you know the complete rule; the rule evaluated at the random observation is $E [X ∣ Y]$ . Being a random variable, it has a distribution, an expectation, a variance — and interrogating those is precisely what the next two sections do.

!Pitfall — E[X | Y] is not E[X] with Y “plugged in”

Two confusions account for most errors with this object. First: $E [X ∣ Y]$ is not obtained by taking the number $E [X]$ and decorating it with a $Y$ — it is a genuinely different object, computed from the conditional distribution, and it varies with $Y$ whenever $X$ and $Y$ are dependent (it collapses to the constant $E [X]$ exactly when knowing $Y$ tells you nothing about the mean of $X$ ). Second: keep $E [X ∣ Y = y]$ (a number, one per $y$ ) distinct from $E [X ∣ Y]$ (a random variable). Identities relate objects of the same type: the tower property below equates two numbers, $E [E [X ∣ Y]]$ and $E [X]$ , by first averaging the random guess over $Y$ . Writing which type each expression has — number or function of $Y$ — catches sign-level blunders before they happen, including the two-envelope trap at the end of this chapter.

The tower property (Adam’s law)#

The first thing to ask of the random variable $E [X ∣ Y]$ is: what is its expectation? The answer is the law of iterated expectations, previewed in joint distributions and proved properly here:

E [E [X ∣ Y]] = E [X] .

✦Proof — swap the order of summation

Discrete case. $E [X ∣ Y]$ is the function $m (Y)$ , so its expectation is a weighted average of the numbers $m (y)$ :

E [m (Y)] = y \sum m (y) P (Y = y) = y \sum x \sum x \frac{P ( X = x , Y = y )}{P ( Y = y )} P (Y = y) = x \sum x y \sum P (X = x, Y = y) = x \sum x P (X = x) = E [X] . ■

The $P (Y = y)$ factors cancel, the sum over $y$ marginalises the joint, and nothing else happens — the theorem is bookkeeping. In words: an average of averages, weighted correctly. Group outcomes by the value of $Y$ , average within each group, then average the group means weighted by group probabilities: you must recover the grand mean, because every outcome is counted exactly once with its original weight. It is the expectation-level version of the law of total probability.

The tower property is the licence behind every “condition on something helpful” computation: to find a hard unconditional expectation, choose a $Y$ that makes the conditional answer easy, then average. First-step analysis below is this strategy industrialised. It also composes: conditioning on more information first and less information second gives $E [E [X ∣ Y, Z] ∣ Y] = E [X ∣ Y]$ — the form that powers every martingale argument in martingales & optional stopping.

Scatter of simulated X, Y pairs with the conditional mean curve E[Y|X=x] drawn through the cloud and the flat dashed line at the unconditional mean E[Y] — E[Y|X] is a function — the best-guess curve threading the cloud, tracking the local mean of Y at each x. The flat dashed line is the single number E[Y]. The tower property is the statement that averaging the curve over the distribution of X lands exactly on the flat line: iterated expectation made visible.

The law of total variance (Eve’s law)#

The second question: how does the variance of $X$ relate to the conditional world? Define the conditional variance $Var (X ∣ Y) = E [X^{2} ∣ Y] - (E [X ∣ Y])^{2}$ — again a random variable, a function of $Y$ . Then:

Var (X) = within-group E [Var (X ∣ Y)] + between-group Var (E [X ∣ Y]) .

✦Proof — expand both second moments with the tower property

Apply the tower property to $X^{2}$ : $E [X^{2}] = E [E [X^{2} ∣ Y]] = E [Var (X ∣ Y) + (E [X ∣ Y])^{2}]$ , using the definition of conditional variance inside. Subtract $(E [X])^{2} = (E [E [X ∣ Y]])^{2}$ (tower again):

Var (X) = E [Var (X ∣ Y)] + (E [(E [X ∣ Y])^{2}] - (E [E [X ∣ Y]])^{2}) = E [Var (X ∣ Y)] + Var (E [X ∣ Y]) . ■

The bracketed term is exactly the variance of the random variable $E [X ∣ Y]$ — the proof only works because rung 3 of the ladder made that a legitimate object. Intuition: total scatter = average scatter within groups + scatter between the group means. Both terms are non-negative, so conditioning never increases expected variance: $E [Var (X ∣ Y)] \leq Var (X)$ — information can only help, on average.

→Quant lens — decomposing PnL variance across regimes

Let $X$ be a strategy’s daily PnL and $R$ the market regime: calm with probability $0.7$ (mean $0.1$ , standard deviation $1$ ) and stressed with probability $0.3$ (mean $- 0.2$ , standard deviation $3$ ). Tower: $E [X] = 0.7 (0.1) + 0.3 (- 0.2) = 0.01$ . Eve: within-group $E [Var (X ∣ R)] = 0.7 (1) + 0.3 (9) = 3.4$ ; between-group $Var (E [X ∣ R]) = 0.7 (0.1 - 0.01)^{2} + 0.3 (- 0.2 - 0.01)^{2} = 0.00567 + 0.01323 = 0.0189$ . Total $Var (X) = 3.4189$ , of which $99.4%$ is within-regime — the risk lives inside the stressed regime’s fat volatility, not in the drift difference between regimes. That is the generic shape of trading risk: regime shifts move the mean a little and the volatility a lot, so hedging the mean exposure to regimes barely dents the variance. The decomposition also explains why mixtures grow fat tails — $X$ here has kurtosis well above 3 despite being conditionally Gaussian — the statistical engine behind regime detection and the vol-mixing story in time-series statistics. The simulation below verifies both identities on a three-regime version.

The best predictor: least squares lives here#

Suppose you must predict $X$ using any function $g (Y)$ whatsoever — linear, polynomial, a lookup table — and you are scored by mean squared error. The minimiser is the conditional expectation:

E [(X - g (Y))^{2}] \geq E [(X - E [X ∣ Y])^{2}] = E [Var (X ∣ Y)] for every g .

✦Proof — the error decomposes like Pythagoras

Write $m (Y) = E [X ∣ Y]$ and split the error: $X - g (Y) = (X - m (Y)) + (m (Y) - g (Y))$ . Square and take expectations; the cross term dies by the tower property, because for any function $h (Y) = m (Y) - g (Y)$ ,

E [(X - m (Y)) h (Y)] = E [h (Y) = 0 E [X - m (Y) Y]] = 0,

since given $Y$ , $h (Y)$ is a constant and $E [X ∣ Y] = m (Y)$ by definition. Hence

E [(X - g (Y))^{2}] = E [(X - m (Y))^{2}] + E [(m (Y) - g (Y))^{2}],

and the second term is non-negative, zero exactly when $g = m$ (almost surely). $■$ Geometrically: the prediction residual $X - m (Y)$ is orthogonal to every function of $Y$ — conditional expectation is a projection, and the display is Pythagoras.

This is why $m (y) = E [Y ∣ X = x]$ is called the regression function: every supervised method that minimises squared error — ordinary least squares in regression & inference, trees, neural networks — is an estimate of this one theoretical object from finite data. OLS restricts $g$ to linear functions and therefore finds the best linear approximation to the curve; when the true $m$ is nonlinear the gap between them is real, predictable, and exploitable (Problem 5 makes it exactly $2/9$ on a toy example). The irreducible floor, $E [Var (X ∣ Y)]$ , is Eve’s within-group term: no model, however flexible, predicts away the scatter that survives conditioning.

First-step analysis: condition on the first move#

Point the tower property at a process: to compute an expected duration, condition on the first step, whose conditional answers are one step closer to done. The memoryless structure that makes this exact is the subject of Markov chains; here are the three canonical computations.

Flips until the first head. Let $E$ be the expected number of fair flips. Condition on flip one: with probability $\frac{1}{2}$ you are done in 1; with probability $\frac{1}{2}$ you spent 1 flip and — by the fresh-start property — face the original problem again:

E = \frac{1}{2} (1) + \frac{1}{2} (1 + E) ⟹ E = 2.

One line replaces the geometric series of common distributions.

✦Worked example — HH takes 6 flips, HT takes 4

HH. Track progress: state 0 (no useful history) with expected remaining time $E_{0}$ , state 1 (last flip was H) with $E_{1}$ . Condition on the next flip in each state:

E_{0} = 1 + \frac{1}{2} E_{1} + \frac{1}{2} E_{0}, E_{1} = 1 + \frac{1}{2} \cdot 0 + \frac{1}{2} E_{0} .

The first equation gives $E_{0} = 2 + E_{1}$ ; substituting the second, $E_{0} = 3 + \frac{1}{2} E_{0}$ , so $E_{0} = 6$ (and $E_{1} = 4$ ).

HT. Same states: $E_{0} = 1 + \frac{1}{2} E_{1} + \frac{1}{2} E_{0}$ again, but now from state 1 a tail finishes and a head keeps you in state 1 — a head can never hurt an HT-hunter:

E_{1} = 1 + \frac{1}{2} \cdot 0 + \frac{1}{2} E_{1} ⟹ E_{1} = 2, E_{0} = 2 + E_{1} = 4.

Why the asymmetry, when both patterns have probability $\frac{1}{4}$ in any two given flips? Look at what failure costs. Hunting HH in state 1, a tail demolishes you back to state 0 — all progress lost. Hunting HT in state 1, the failing flip is a head, which leaves you exactly where you were: progress toward HT never resets. Self-overlapping patterns recycle their near-misses into rivals’ progress, not their own — the same overlap arithmetic ( $2^{2} + 2 = 6$ vs $2^{2} = 4$ ) that the betting-team martingale of martingales & optional stopping reads off ABRACADABRA. First-step analysis and the martingale argument are two roads to one truth.

Random sums: Wald’s identity and Eve on N#

Many quantities are sums with a random number of terms: total insurance claims in a year, total PnL over a random number of trades, total offspring in a branching process. Let $X_{1}, X_{2}, \dots$ be i.i.d. with mean $μ$ and variance $σ^{2}$ , let $N$ be a non-negative integer random variable independent of the $X_{i}$ , and set $S_{N} = X_{1} + \dots + X_{N}$ . Condition on $N$ : $E [S_{N} ∣ N] = N μ$ and $Var (S_{N} ∣ N) = N σ^{2}$ , so tower and Eve give

E [S_{N}] = E [N] μ, Var (S_{N}) = E [N] σ^{2} + μ^{2} Var (N) .

The mean formula is Wald’s identity in its independent- $N$ form; the remarkable upgrade — it survives when $N$ is a stopping time that peeks at the $X_{i}$ as they arrive — is proved in martingales & optional stopping. The variance formula is pure Eve: within-group scatter $E [N] σ^{2}$ from the terms themselves, between-group scatter $μ^{2} Var (N)$ from not knowing how many there are. Note the second term vanishes when $μ = 0$ : for a zero-edge book, uncertainty in the number of trades adds nothing to PnL variance. For Poisson $N$ with rate $λ$ , both pieces merge into the compound-Poisson formula $Var (S_{N}) = λ E [X^{2}]$ (Poisson processes), worked with numbers in Problem 2.

A closing teaser on misusingthe machinery. Two envelopes, one holding twice the other; you hold one. The seductive computation — “the other contains $2 x$ or $x /2$ with probability $\frac{1}{2}$ each, so its conditional expectation is $1.25 x$ , so always switch” — silently assumes that given your envelope contains $x$ , both cases are equally likely for every $x$ , which no genuine prior on the amounts can satisfy. It is a type error on the ladder: a statement about $E [other ∣ yours = x]$ that was never derived from any joint distribution. The full dissection, including priors under which switching genuinely is right for some $x$ , is in probability paradoxes.

tower_and_eve.py

import numpy as np

rng = np.random.default_rng(21)
n = 1_000_000

# Daily PnL from a 3-regime mixture: calm / choppy / crisis
p     = np.array([0.60, 0.30, 0.10])   # regime probabilities
mu    = np.array([0.08, 0.00, -0.50])  # conditional means E[X|R]
sigma = np.array([0.60, 1.20, 3.00])   # conditional vols  sd(X|R)

regime = rng.choice(3, size=n, p=p)
pnl = rng.normal(mu[regime], sigma[regime])

# Tower property: E[X] = E[ E[X|R] ] = sum_k p_k mu_k
lhs_mean = pnl.mean()
rhs_mean = p @ mu
print(f"tower:  E[X] sim = {lhs_mean:+.4f}   E[E[X|R]] = {rhs_mean:+.4f}")

# Eve's law: Var(X) = E[Var(X|R)] + Var(E[X|R])
within  = p @ sigma**2                 # E[Var(X|R)]
between = p @ (mu - rhs_mean)**2       # Var(E[X|R])
print(f"eve:    Var(X) sim = {pnl.var():.4f}   "
      f"within {within:.4f} + between {between:.4f} = {within + between:.4f}")

# Same decomposition estimated from the sample, group by group
p_hat  = np.bincount(regime) / n
m_hat  = np.array([pnl[regime == k].mean() for k in range(3)])
v_hat  = np.array([pnl[regime == k].var()  for k in range(3)])
print(f"sample: within {p_hat @ v_hat:.4f} + "
      f"between {p_hat @ (m_hat - lhs_mean)**2:.4f}")
# Theory: E[X] = -0.0020, Var(X) = 1.5480 + 0.0288 = 1.5768 --
# 98% of the variance is within-regime, and the mixture's kurtosis
# (try from scipy.stats import kurtosis) is far above Gaussian.

Practice problems#

Five problems that test whether the ladder, the two laws, and first-step conditioning are actually usable — each yields to writing down what is being conditioned on.

→Problem 1 — two consecutive sixes

Roll a fair die repeatedly. What is the expected number of rolls until you first see two sixes in a row?

Solution. First-step analysis with two states: $E_{0}$ (last roll not a six) and $E_{1}$ (last roll a six). Conditioning on the next roll:

E_{0} = 1 + \frac{1}{6} E_{1} + \frac{5}{6} E_{0}, E_{1} = 1 + \frac{1}{6} \cdot 0 + \frac{5}{6} E_{0} .

The first gives $E_{0} = 6 + E_{1}$ ; substituting the second, $E_{0} = 7 + \frac{5}{6} E_{0}$ , so $E_{0} = 42$ (and $E_{1} = 36$ ). The general pattern for a success probability $p$ : two in a row costs $p^{- 1} + p^{- 2} = 6 + 36 = 42$ , matching the overlap rule (the pattern “66” overlaps itself in “6”). Lesson: name the progress states, condition on one step, and the recursion does the rest — no distributions required.

→Problem 2 — total insurance claims

Claims arrive in a year as $N \sim Poisson (λ = 20)$ , independent of claim sizes $X_{i} \sim Exponential$ with mean $5$ (in thousands). Find the mean and variance of the total payout $S_{N}$ .

Solution. Wald: $E [S_{N}] = E [N] μ = 20 \times 5 = 100$ . Eve with $σ^{2} = 25$ (exponential: variance = mean²) and $Var (N) = 20$ : $Var (S_{N}) = 20 \times 25 + 25 \times 20 = 1000$ — equivalently the compound-Poisson shortcut $λ E [X^{2}] = 20 \times 2 (5^{2}) = 1000$ , standard deviation $1000 \approx 31.6$ . Note the two variance sources contribute equally here ( $500 + 500$ ): random claim count is as risky as random claim size. Lesson: never bolt $Var (N)$ onto $σ^{2}$ by guesswork — Eve dictates the exact weights.

→Problem 3 — a die, then coins: two-stage conditioning

Roll a fair die to get $N$ , then flip $N$ fair coins. Find the mean and variance of the number of heads $H$ .

Solution. Given $N$ , $H \sim Binomial (N, \frac{1}{2})$ , so $E [H ∣ N] = N /2$ and $Var (H ∣ N) = N /4$ . Tower: $E [H] = E [N] /2 = 3.5/2 = 1.75$ . Eve:

Var (H) = E [\frac{N}{4}] + Var (\frac{N}{2}) = \frac{3.5}{4} + \frac{35/12}{4} = \frac{42}{48} + \frac{35}{48} = \frac{77}{48} \approx 1.604.

Sanity checks: larger than a fixed- $N$ binomial’s $3.5/4 = 0.875$ (die uncertainty adds between-group variance), and both pieces are visibly non-negative. Lesson: two-stage experiments are not harder than one-stage ones — they are one conditioning away from being routine.

→Problem 4 — the larger of two dice, by conditioning

Roll two fair dice. Compute $E [max]$ by conditioning on the first die.

Solution. Given the first die shows $i$ , the max is $i$ if the second lands in ${1, \dots, i}$ and otherwise equals the second die: $E [max ∣ X_{1} = i] = \frac{1}{6} (i \cdot i + \sum_{j > i} j)$ . The six conditional answers are $\frac{21}{6}, \frac{22}{6}, \frac{24}{6}, \frac{27}{6}, \frac{31}{6}, \frac{36}{6}$ — a genuinely varying function of the first roll, i.e. the random variable $E [max ∣ X_{1}]$ in the flesh. Tower-average them:

E [max] = \frac{21 + 22 + 24 + 27 + 31 + 36}{36} = \frac{161}{36} \approx 4.47,

agreeing with the tail-sum route via $P (max \leq k) = (k /6)^{2}$ from order statistics. Lesson: when a joint quantity resists direct attack, condition on one coordinate — the conditional answers are one-dimensional, and the tower property stitches them back together.

→Problem 5 — beating every linear predictor

Let $X$ be uniform on ${- 1, 0, 1}$ and $Y = X^{2}$ . Find the best predictor of $Y$ from $X$ , the best linear predictor, and both mean squared errors.

Solution. The best predictor is $E [Y ∣ X] = X^{2}$ — it predicts perfectly, MSE $= 0$ . The best linear predictor $a + b X$ needs $b = Cov (X, Y) / Var (X)$ , and $Cov (X, Y) = E [X^{3}] - E [X] E [X^{2}] = 0$ by symmetry — so the best line is the flat constant $E [Y] = \frac{2}{3}$ , with MSE $= Var (Y) = \frac{2}{3} - \frac{4}{9} = \frac{2}{9} \approx 0.222$ . Here $X$ and $Y$ are uncorrelated yet completely dependent: correlation sees only the linear shadow of the regression function. Lesson: “no correlation” never means “no predictability” — a nonlinear $E [Y ∣ X]$ can carry an entire signal that every linear model scores as zero, which is exactly the gap nonlinear features exist to close in regression & inference.

Next: the other great labour-saving principle — arguments that finish before computation starts because relabelling cannot change an answer: exchangeability, symmetric sampling, and why uniform random positions are so often the whole solution. Symmetry & exchangeability.