Statistical Independence

Two events are independent when knowing that one happened does not change the probability of the other.

Two events are independent if knowing that one of them happened tells you nothing about whether the other happened. Two tosses of a fair coin are the standard example: the first toss has no influence on the second. Independence is one of the most important assumptions in statistics. It is what lets us multiply probabilities, treat the observations in a sample as separate pieces of information, and derive most of the formulas used in inference.

Intuition

Suppose you are told that it rained in Paris today. Does that change your estimate of the probability that a die you are about to roll shows a 6? No: the two have nothing to do with each other. Now suppose you are told that a randomly chosen card is red. That does change the probability that it is a heart, from $1/4$ to $1/2$ . The first pair of events is independent; the second is not.

In terms of conditional probability, independence of $A$ and $B$ means that conditioning on $B$ leaves the probability of $A$ unchanged:

P(A \mid B) = P(A).

Definition

Two events $A$ and $B$ are independent if

P(A \cap B) = P(A)\, P(B).

In words, the probability that both happen is the product of their separate probabilities.

When $P(B) > 0$ , this is equivalent to $P(A \mid B) = P(A)$ . To see why, divide both sides of the definition by $P(B)$ : the left side becomes $P(A \cap B)/P(B) = P(A \mid B)$ , and the right side becomes $P(A)$ . Similarly, when $P(A) > 0$ , it is equivalent to $P(B \mid A) = P(B)$ , so independence is symmetric: if $A$ tells you nothing about $B$ , then $B$ tells you nothing about $A$ .

The product form is used as the definition because it is symmetric, needs no condition like $P(B) > 0$ , and is the form used in calculations.

Some useful facts:

If $A$ and $B$ are independent, so are $A$ and $B^c$ , $A^c$ and $B$ , and $A^c$ and $B^c$ . Knowing that $B$ did not happen is just as uninformative as knowing that it did.
Independence is usually an assumption justified by how the experiment works (separate coin tosses, people sampled at random), not something read off from the event descriptions.

Worked example

A company records the handedness of 400 employees and whether they drink coffee.

	Left-handed	Right-handed	Total
Drinks coffee	24	216	240
No coffee	16	144	160
Total	40	360	400

Pick an employee at random. Let $L$ be “left-handed” and $K$ be “drinks coffee”. Are these events independent?

Step 1: the separate probabilities. $P(L) = 40/400 = 0.10$ and $P(K) = 240/400 = 0.60$ .

Step 2: the joint probability. $P(L \cap K) = 24/400 = 0.06$ .

Step 3: compare with the product. $P(L)\,P(K) = 0.10 \times 0.60 = 0.06$ . The two agree, so $L$ and $K$ are independent.

The same check with conditional probabilities. Among coffee drinkers, $P(L \mid K) = 24/240 = 0.10$ ; among non-drinkers, $P(L \mid K^c) = 16/160 = 0.10$ . The proportion of left-handers is the same in both groups and equal to the overall $0.10$ .

A dependent table for contrast. In the dog-and-cat survey in the conditional probability article, $P(\text{cat}) = 0.35$ but $P(\text{cat} \mid \text{dog}) = 0.375$ . Since these differ, owning a cat and owning a dog are not independent in that survey; equivalently, $P(\text{cat} \cap \text{dog}) = 0.15$ differs from $0.35 \times 0.40 = 0.14$ .

In real data, the observed proportions will almost never match the product exactly, even when the underlying events are independent, because of random sampling variation. Deciding whether a difference like $0.15$ versus $0.14$ reflects real dependence or just chance is a question for hypothesis testing.

Independent is not the same as mutually exclusive

This is the most common confusion about independence. Mutually exclusive (disjoint) events cannot both happen: $A \cap B = \varnothing$ . Independent events can happen together, and do so exactly as often as chance alone predicts.

In fact, two mutually exclusive events that each have positive probability are always dependent. If $A$ and $B$ are disjoint, then $P(A \cap B) = 0$ , but $P(A)\,P(B) > 0$ . Knowing that $A$ happened tells you for certain that $B$ did not, which is the strongest possible information.

For one roll of a die, $A = \{1, 2\}$ and $B = \{5, 6\}$ are mutually exclusive. Here $P(A \cap B) = 0$ while $P(A)\,P(B) = \tfrac{1}{3} \cdot \tfrac{1}{3} = \tfrac{1}{9}$ , so they are not independent.

More than two events

Events $A_1, A_2, \dots, A_n$ are mutually independent if the product rule holds for every subcollection of them, not only for pairs. For three events $A$ , $B$ , $C$ , this means all four of the following:

\begin{aligned} P(A \cap B) &= P(A)P(B), & P(A \cap C) &= P(A)P(C), \\ P(B \cap C) &= P(B)P(C), & P(A \cap B \cap C) &= P(A)P(B)P(C). \end{aligned}

If only the three pairwise conditions hold, the events are pairwise independent. Pairwise independence does not imply mutual independence.

Example. Toss a fair coin twice; the four outcomes HH, HT, TH, TT each have probability $1/4$ . Let

$A$ = “the first toss is heads” = $\{\text{HH}, \text{HT}\}$ ,
$B$ = “the second toss is heads” = $\{\text{HH}, \text{TH}\}$ ,
$C$ = “the two tosses agree” = $\{\text{HH}, \text{TT}\}$ .

Each event has probability $1/2$ . Each pair overlaps in exactly one outcome, HH, so each pairwise intersection has probability $1/4 = \tfrac12 \cdot \tfrac12$ : the events are pairwise independent. But

P(A \cap B \cap C) = P(\{\text{HH}\}) = \tfrac{1}{4} \ne \tfrac{1}{8} = P(A)P(B)P(C).

The reason is that any two of the events determine the third: if both tosses are heads, they certainly agree. When people say “independent” without qualification about several events or several observations, they mean mutual independence.

Independence of random variables

The idea extends from events to random variables. Two random variables $X$ and $Y$ are independent if every event about $X$ is independent of every event about $Y$ :

P(X \le x \text{ and } Y \le y) = P(X \le x)\, P(Y \le y) \quad \text{for all } x, y.

For discrete variables this is equivalent to $P(X = x, Y = y) = P(X = x)\,P(Y = y)$ for all values $x$ and $y$ . The results of two separate dice rolls are independent random variables.

Independence of random variables is the basis of the familiar phrase “independent and identically distributed” (i.i.d.) for a random sample. It is also why the variance of a sum of independent variables is the sum of their variances. Independent variables always have zero covariance, but zero covariance does not imply independence.

Common misunderstandings

“Disjoint events are independent.” The opposite is true: disjoint events with positive probabilities are dependent. See above.

“If two events are dependent, one must cause the other.” Dependence is a statement about probabilities, not about mechanisms. Two events can be dependent because both are influenced by a third factor: ice-cream sales and sunburn are related because both rise in sunny weather.

“A coin that has landed tails five times in a row is more likely to land heads next.” If the tosses are independent, the probability of heads on the next toss is still $1/2$ . This is the gambler’s fallacy.

“If every pair is independent, the whole collection is independent.” Not in general; see the two-coin example above.