Part 4 · Probability and statistical physics

4.1 Probability and random variables

Source: Postorino, week 4 slides, pp. 13–15, quoting De Vincenzi’s notes, ch. 5; Sciubba’s notes

Objectives

Vocabulary. Classical, frequentist and axiomatic probability; random variable; probability distribution; probability density; cumulative distribution; expected value; variance; standard deviation.
Theorem. Kolmogorov’s three axioms, and normalisation $\sum_{i} p_{i} = 1$ or $\int f = 1$ .
Recipe. Mean and variance of a distribution: $μ = E [X]$ , $σ^{2} = E [X^{2}] - μ^{2}$ .
Picture. Probabilities are areas under the density; the cumulative is the running area, from 0 to 1.

The slides’ boxes are excerpts from De Vincenzi’s notes, ch. 5, posted on Moodle: the equation numbers below are his. De Vincenzi’s notation: $E [\cdot]$ for the expected value, $V ar [\cdot]$ for the variance, $\overline{A}$ for the complement of $A$ .

Definitions of probability

A probability is a quantitative measure of how possible an event is: by convention a number between 0 (the impossible event) and 1 (the certain event). A perfectly symmetric coin gives heads and tails with probability 0.5 each. The concept is intuitive, but defining it is not; the slides give three definitions.

Definition 4.1.1Classical probability (Laplace, 1812)slides w4, p. 13

The probability of an event is the ratio of the number of favourable cases to the number of possible cases, provided the latter are equally possible.

It fits ideal or nearly ideal situations: a perfectly cubic die, a card from a well-shuffled deck, a lottery draw. Where it applies, combinatorics does the counting.
It is circular: “equally possible” means “equally probable”, so it uses probability to define probability.
It fails when the set of possible cases is infinite.

Definition 4.1.2Frequentist probability (von Mises, 1920)slides w4, p. 13

Take a repeatable experiment and a “favourable” outcome. The frequency is $k / n$ , the number of favourable outcomes $k$ over the number of trials $n$ . The probability is the limit to which the frequency tends as the number of trials goes to infinity: $P = n \to \infty lim \frac{k}{n} . (5.1)$

It is used in insurance and quality control.
It sets the right relation between frequency, measured a posteriori, and probability. In the classical view probability is fixed a priori.
It needs trials repeatable without limit under identical conditions; real experiments meet both only approximately.
It says nothing about events that can’t be repeated: that a given star explodes in the next 30 days, that it rains in Rome tomorrow.

Definition 4.1.3Axiomatic probability (Kolmogorov, 1933)slides w4, p. 13

Postulate a space $Ω$ containing all the elementary events ${E_{i}}$ that can occur. The probability $P (E)$ is a real function on the events of $Ω$ obeying three axioms:

Positivity: $P (E_{i}) \geq 0$ .
Certainty (unitarity): the certain event, the whole sample space, has $P (Ω) = 1$ .
Union: if $E_{i}$ and $E_{j}$ are mutually exclusive (incompatible), $P (E_{i} \cup E_{j}) = P (E_{i}) + P (E_{j})$ , with $E_{i} \cap E_{j} = \emptyset$ .

Every rule of probability follows from these three. But the theory gives no operational way of assigning probabilities: the values must come from outside, from symmetry (the classical way), from frequencies, or from judgement.

Corollary 4.1.4The complementbeyond the lectures

$P (\overline{A}) = 1 - P (A)$ ; in particular $P (\emptyset) = 0$ .

Proofbeyond the lectures

$A$ and $\overline{A}$ are mutually exclusive and together make $Ω$ , so by axioms 3 and 2, $P (A) + P (\overline{A}) = 1$ . Take $A = Ω$ , $\overline{A} = \emptyset$ .

Loading the demo…

Figure 4.1.1 The sum of two dice. The bars are the classical probabilities

k /36

; the

dots

are frequencies from simulated rolls. Roll 10, 100 or 1000 at a time: the frequencies settle onto the probabilities, the frequentist definition (5.1) at work. Click a range of values to add up its probability.

Extra reading: subjective probability and rules for combining events

De Vincenzi, pp. 3–8, and Sciubba, pp. 1–7. Not examined.

Subjective definition (de Finetti, 1937). Probability is the degree of belief a coherent individual assigns to an event, given their information. Operationally, it is the price $p$ they consider fair to pay to receive 1 if the event happens; coherent means they would equally accept the reverse bet. It depends on information: someone who glimpses a side face of a die after it lands rates a “6” at 0 or 1/4, not 1/6.
Union of overlapping events. $P (A + B) = P (A) + P (B) - P (A B), (5.2)$ with $A + B$ the union and $A B$ the intersection. Proof: $A + B = A + \overline{A} B$ is a union of disjoint events, and $P (\overline{A} B) = P (B) - P (A B)$ .
Conditional probability. $P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )} . (5.4)$ Knowing $B$ happened shrinks the sample space to $B$ . Die: $P (3 ∣ odd) = \frac{1/6}{1/2} = \frac{1}{3}$ . Two dead bulbs among 7, tested one by one: $P (first two dead) = \frac{2}{7} \cdot \frac{1}{6} = \frac{1}{21}$ .
Independence. $P (A ∣ B) = P (A)$ , so $P (A B) = P (A) P (B)$ : for independent events probabilities multiply. Three roulette wins in a row: $(1/37)^{3} = 1.97 \times 1 0^{- 5}$ .
Bayes’ theorem. $P (A ∣ B) P (B) = P (B ∣ A) P (A) \Rightarrow P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B ∣ A ) P ( A ) + P ( B ∣ A ) P ( A )} . (5.7)$ A normal coin and a two-headed coin; one is picked at random and shows heads: $P (normal ∣ heads) = \frac{1/4}{3/4} = \frac{1}{3}$ . A test that catches 99.7 % of infections with 7.4 % false positives, in a population 0.5 % infected: a positive result means infection with probability only 6.3 %, because the infected are so rare.
Law of large numbers (Sciubba, pp. 5–6). As coin tosses accumulate, the fraction of heads fluctuates less and settles near 0.5. Bernoulli’s theorem: for any $ε > 0$ , $lim_{N \to \infty} P (∣ p - k / N ∣ < ε) = 1$ . The frequency doesn’t converge to $p$ in the strict sense; it becomes ever more probable that it is close.

Discrete random variables

A random variable (variabile aleatoria, or casuale, stocastica) is one that can’t be given a definite value, only a range of values with probabilities. Its probability distribution says with what probability it takes each value.

Example 4.1.5The sum of two diceslides w4, p. 14

$X$ = the sum of two dice takes the values 2 to 12, from $6 \times 6 = 36$ equally likely ordered pairs.

$X = 2$ and $X = 12$ arise one way each: $P (2) = P (12) = 1/36 = 0.028$ . (De Vincenzi prints the first as “ $P (1)$ ”, a slip.)
$X = 7$ arises from 6 pairs, (1,6), (2,5), (3,4), (4,3), (5,2), (6,1): $P (7) = 6/36 = 0.167$ .
The histogram is a triangle: $k /36$ with $k = 1, 2, 3, 4, 5, 6, 5, 4, 3, 2, 1$ .

Definition 4.1.6Probability distribution of a discrete variableslides w4, p. 14

For a discrete random variable $X$ that can take the values $x_{1}, x_{2}, \dots, x_{n}$ (possibly $n = \infty$ ), its probability distribution is the set of probabilities $p_{i}$ of the $x_{i}$ : $P (x_{i}) = p_{i} . (5.8)$ If ${x_{1}, \dots, x_{n}}$ is a complete set of mutually exclusive events, so that $X$ can take no other values, the normalisation condition holds: $i \sum p_{i} = 1.$

Normalisation is axioms 2 and 3 together: “ $X$ takes some value” is the certain event, and it is the union of the mutually exclusive events “ $X = x_{i}$ ” beyond the lectures.

Continuous random variables

A continuous variable $X$ takes uncountably many values $x$ . So, unlike the discrete case, the probability that it takes exactly one particular value is zero. What makes sense is the probability that it falls in an interval, $P (x < X < x + Δ x)$ ; for an infinitesimal interval this is infinitesimal too: $d P = \frac{d P}{d x} d x = f (x) d x .$

Definition 4.1.7Probability density functionEssentialslides w4, p. 14

$f (x) = d P / d x$ is the probability density function (pdf). It has two properties:

it is positive or zero over the whole domain: $f (x) \geq 0$ for every $x \in Ω$ ;
it is normalised: $\int_{Ω} f (x) d x = 1$ .

Probabilities are areas under it: $P (x < B) = \int_{- \infty}^{B} f (x) d x, P (A < x < B) = \int_{A}^{B} f (x) d x .$

$f$ is a probability per unit $x$ , not a probability, so it can exceed 1 beyond the lectures.

Definition 4.1.8Cumulative distribution functionslides w4, p. 15

The probability that the variable takes any value below a given $X$ : $F (X) \equiv P (x < X) = \int_{- \infty}^{X} f (x) d x . (5.9)$

De Vincenzi’s (5.9) writes $F (x)$ with upper limit $X$ , mixing the two letters; the slide’s figure writes it cleanly as $F (x) = \int_{- \infty}^{x} f (x^{'}) d x^{'}$ . The figure: on the left a pdf, skewed with a long tail to the right, with a thin strip of width $d x$ of area $f (x) d x$ ; on the right its cumulative, rising from 0 to 1 (certainty).

Proposition 4.1.9Properties of the cumulativebeyond the lectures

$F$ never decreases; $F (- \infty) = 0$ , $F (+ \infty) = 1$ ; $P (A < x < B) = F (B) - F (A)$ ; and $f = d F / d x$ : the pdf is the slope of the cumulative.

Proofbeyond the lectures

$f \geq 0$ , so the running integral can only grow. The limits are the empty interval and normalisation. The difference of two running integrals is the integral between them. The last is the fundamental theorem of calculus.

Loading the demo…

Figure 4.1.2 A probability density

f (x)

and its cumulative

F (x)

. Drag the ends

A

and

B

: the shaded area is

P (A < x < B)

, which equals

F (B) - F (A)

, the rise of the cumulative between them. The

mean

and

μ \pm σ

are marked. Try the distributions: the exponential is skewed like the slide’s, the Gaussian symmetric.

Expected value and variance

A distribution is fully determined by its form, but a few parameters summarise it: where its centre is (the mean), how spread out it is (the variance), how symmetric it is, and its shape. The tool for all of them is the expected value.

Definition 4.1.10Expected valueslides w4, p. 15

For any function $g (X)$ , $E [g (X)] = i \sum g (x_{i}) p_{i} (discrete), E [g (X)] = \int_{Ω} g (x) f (x) d x (continuous) . (5.10)$ It is the probability-weighted average of $g$ .

Proposition 4.1.11Properties of the expected valueDe Vincenzi, p. 10

Linearity: $E [a g (X) + b h (X)] = a E [g (X)] + b E [h (X)]$ , since sums and integrals are linear.
The expected value of a constant is the constant: $E [c] = c, (5.11)$ by normalisation. So $E [E [x]] = E [x]$ .

Definition 4.1.12Meanslides w4, p. 15

The mean (valore medio) is the case $g (x) = x$ : $E (X) \equiv μ = i \sum x_{i} p_{i} (discrete), E (X) \equiv μ = \int_{Ω} x f (x) d x (continuous) . (5.12)$

Theorem 4.1.13Varianceslides w4, p. 15

The expected value of the squared deviation $(X - μ)^{2}$ is the variance: $V ar [X] \equiv σ^{2} = E [(X - μ)^{2}] = E [X^{2}] - μ^{2} . (5.13)$ The variance is the expected value of the square minus the square of the expected value. Its square root $σ$ is the standard deviation, in the same units as $X$ .

Proof

$E [(X - μ)^{2}] = E [X^{2} - 2 μ X + μ^{2}] = E [X^{2}] - 2 μ E [X] + μ^{2} = E [X^{2}] - 2 μ^{2} + μ^{2}$ , by linearity, $E [X] = μ$ and $E [μ^{2}] = μ^{2}$ .

Proposition 4.1.14Variance is not linearDe Vincenzi, p. 11

For constants $a$ and $b$ : $V ar [a X] = a^{2} V ar [X], (5.14)$ $V ar [a X + b] = a^{2} V ar [X] . (5.15)$

Proofbeyond the lectures

De Vincenzi leaves it as an exercise. The mean of $a X + b$ is $a μ + b$ , so its deviation is $a (X - μ)$ ; squared, $a^{2} (X - μ)^{2}$ , with expected value $a^{2} σ^{2}$ . Shifting by $b$ moves the whole distribution without spreading it.

The mean says roughly where the probability is concentrated, and $σ$ how wide that region is. Both are rough guides, and for some distributions they don’t even exist (the Cauchy, below).

Example 4.1.15One diebeyond the lectures

$μ = \frac{1}{6} (1 + \dots + 6) = \frac{7}{2}$ ; $E [X^{2}] = \frac{1}{6} (1 + 4 + \dots + 36) = \frac{91}{6}$ . So $σ^{2} = \frac{91}{6} - \frac{49}{4} = \frac{35}{12} = 2.92$ and $σ = 1.71$ .

Extra reading: distributions, moments, several variables, Chebyshev

De Vincenzi, pp. 12–36, and Sciubba, pp. 8–31. Not examined.

Discrete distributions

Bernoulli: one trial, $P (1) = p$ , $P (0) = q = 1 - p$ ; $E = p$ , $V ar = pq$ .
Binomial: $k$ successes in $N$ independent trials, $B_{N} (k) = (k N) p^{k} q^{N - k}$ . The factor $p^{k} q^{N - k}$ is one order; the binomial coefficient counts the orders. $\sum_{k} B_{N} (k) = (p + q)^{N} = 1$ , $μ = Np$ , $V ar = Npq$ .
Poisson: independent events at a constant rate, counted in an interval, $P_{μ} (k) = \frac{μ ^{k}}{k !} e^{- μ}$ . Mean and variance are both $μ$ . It is the binomial’s limit for $N \to \infty$ , $p \to 0$ with $Np = μ$ fixed. Cosmic rays at $1.2 \times 1 0^{2}$ m⁻²s⁻¹ on a $1 0^{2}$ cm² detector for 6 s: $μ = 7.2$ , $σ = 2.7$ .

Continuous distributions

Uniform on $(a, b)$ : $f = \frac{1}{b - a}$ , $μ = \frac{a + b}{2}$ , $V ar = \frac{( b - a ) ^{2}}{12}$ . A digital display’s last digit gives an uncertainty of $1/ 12$ of that digit.
Gaussian: $f = \frac{1}{2 π σ} e^{- (x - μ)^{2} /2 σ^{2}}$ . With $z = (x - μ) / σ$ , $P (a < x < b) = Φ (z_{b}) - Φ (z_{a})$ , $Φ (x) = \frac{1}{2} [1 + erf (x / 2)]$ . Within $1 σ$ , $2 σ$ , $3 σ$ : 68.3 %, 95.4 %, 99.7 %.
Exponential: $f = \frac{1}{τ} e^{- t / τ}$ , mean $τ$ , variance $τ^{2}$ , $F = 1 - e^{- t / τ}$ : the waiting time to the next event of a Poisson process, and memoryless.
Student’s $t$ : the sample mean of a normal variable with estimated variance; wider than a Gaussian, tending to one as the degrees of freedom grow.
Cauchy (Breit–Wigner, Lorentzian): $f = \frac{Γ}{π} \frac{1}{( x - x _{0} ) ^{2} + Γ ^{2}}$ . No mean and no variance: the integrals diverge. It is the resonance lineshape of Proposition 1.4.7.
$χ^{2}$ with $n$ degrees of freedom: the sum of squares of $n$ standard normals; mean $n$ , variance $2 n$ .

Moments. Raw $m_{n}^{'} = E [x^{n}]$ , central $m_{n} = E [(x - μ)^{n}]$ . Skewness $m_{3} / σ^{3}$ , kurtosis $m_{4} / σ^{4} - 3$ ; mode (peak) and median (half the probability on each side).

Several variables. Marginals $\int f (x, y) d y$ ; covariance $E [(x - μ_{x}) (y - μ_{y})]$ and correlation $ρ \in [- 1, 1]$ . $V ar [a X + bY] = a^{2} σ_{X}^{2} + b^{2} σ_{Y}^{2} + 2 ab Cov$ , so the mean of $N$ independent measurements has variance $σ^{2} / N$ . Propagation of uncertainty: $σ^{2} (Y) \approx \sum_{i} (\partial g / \partial X_{i})^{2} σ^{2} (X_{i})$ . Change of variables: $f_{Y} (y) = f_{X} (x) ∣ d x / d y ∣$ .

Chebyshev’s inequality (Sciubba, pp. 15–16): for any distribution, $P (∣ x - m ∣ \leq K σ) \geq 1 - 1/ K^{2}$ ; at least 75 % within $2 σ$ , 88.9 % within $3 σ$ , whatever the distribution.

Exercises

Exercise 4.1.1beyond the lectures

A pdf is $f (x) = c x^{2}$ on $[0, 3]$ and 0 elsewhere. Find $c$ , $F (x)$ , $P (1 < x < 2)$ , $μ$ and $σ^{2}$ .

Solution

$\int_{0}^{3} c x^{2} d x = 9 c = 1$ , so $c = 1/9$ . $F (x) = x^{3} /27$ on $[0, 3]$ . $P (1 < x < 2) = (8 - 1) /27 = 7/27$ . $μ = \int_{0}^{3} x^{3} /9 d x = 9/4$ . $E [x^{2}] = \int_{0}^{3} x^{4} /9 d x = 27/5$ , so $σ^{2} = 27/5 - 81/16 = 0.3375$ .

Exercise 4.1.2beyond the lectures

Prove $V ar [a X + b] = a^{2} V ar [X]$ from (5.13).

Solution

$E [(a X + b)^{2}] - (a μ + b)^{2} = a^{2} E [X^{2}] + 2 ab μ + b^{2} - a^{2} μ^{2} - 2 ab μ - b^{2} = a^{2} (E [X^{2}] - μ^{2})$ .

Exercise 4.1.3beyond the lectures

Why is the probability of a continuous variable taking exactly $x_{0}$ zero, even though $f (x_{0}) > 0$ ?

Solution

$P (x_{0} < X < x_{0} + Δ x) \approx f (x_{0}) Δ x \to 0$ as $Δ x \to 0$ . Only intervals have non-zero probability: the density is probability per unit length.

Exercise 4.1.4beyond the lectures

Find the mean and variance of the sum of two dice.

Solution

By symmetry $μ = 7$ . $E [X^{2}] = \sum k^{2} P (k) = \frac{1}{36} (4 + 18 + 48 + 100 + 180 + 294 + 320 + 324 + 300 + 242 + 144) = \frac{1974}{36} = 54.83$ , so $σ^{2} = 54.83 - 49 = 5.83 = 2 \cdot \frac{35}{12}$ : twice one die’s variance, as for independent variables.

Exercise 4.1.5beyond the lectures

For the exponential density $f (t) = \frac{1}{τ} e^{- t / τ}$ , $t \geq 0$ , find $F (t)$ , the mean, and the probability of waiting longer than $2 τ$ .

Solution

$F (t) = 1 - e^{- t / τ}$ . $μ = \int_{0}^{\infty} \frac{t}{τ} e^{- t / τ} d t = τ$ . $P (t > 2 τ) = 1 - F (2 τ) = e^{- 2} = 13.5%$ .