Introduction to Probability

The basic language of probability, namely experiments, sample spaces, events, and the three axioms every probability must obey.

Probability is a number between 0 and 1 that measures how likely something is to happen: 0 means it cannot happen, 1 means it is certain, and values in between express degrees of likelihood. Probability is the mathematical language for uncertainty, and it is the foundation on which all of statistics is built: to draw conclusions from data that could have come out differently, we first need a precise way to talk about chance.

Intuition

Roll an ordinary six-sided die. Before it lands you do not know the result, but you do know two things: the result will be one of 1, 2, 3, 4, 5, or 6, and (for a fair die) no face is favoured over another. Probability turns this knowledge into numbers. The chance of rolling a 6 is $1/6$; the chance of rolling an even number is $3/6 = 1/2$, because three of the six equally likely faces are even.

Everything in this article is a careful version of that reasoning. We need three ingredients:

1. a list of everything that could happen,
2. a way of describing the things we are interested in (“an even number”),
3. a rule that assigns each of those things a number between 0 and 1.

Experiments, outcomes, and the sample space

A random experiment is any process whose result is not known in advance: tossing a coin, rolling a die, measuring tomorrow’s rainfall, choosing a person at random from a population.

Each possible result is an outcome. The set of all possible outcomes is the sample space, written $\Omega$ (the Greek capital letter omega).

• Tossing a coin: $\Omega = \{\text{H}, \text{T}\}$.
• Rolling a die: $\Omega = \{1, 2, 3, 4, 5, 6\}$.
• Tossing a coin twice: $\Omega = \{\text{HH}, \text{HT}, \text{TH}, \text{TT}\}$.
• Measuring a waiting time in minutes: $\Omega = [0, \infty)$, all non-negative real numbers.

The first three sample spaces are finite. The last one is continuous: it contains infinitely many outcomes, and that difference will matter later.

Events

An event is a set of outcomes, that is, a subset of $\Omega$. We say the event occurs if the outcome of the experiment belongs to it. Events are usually named with capital letters such as $A$ and $B$.

For one roll of a die:

• “an even number” is $A = \{2, 4, 6\}$;
• “at least 5” is $B = \{5, 6\}$;
• “a 6” is the single-outcome event $\{6\}$.

Because events are sets, we can combine them with the usual set operations. Each has a plain-English reading.

Two events are mutually exclusive (or disjoint) if they cannot both happen, that is, $A \cap B = \varnothing$, where $\varnothing$ is the empty set. “Roll a 1” and “roll an even number” are mutually exclusive.

The axioms of probability

A probability $P$ assigns a number $P(A)$ to each event $A$. In 1933 Andrey Kolmogorov showed that the whole theory can be built on three rules, now called the axioms of probability:

1. Non-negativity. For every event $A$, $P(A) \ge 0$.
2. Normalization. $P(\Omega) = 1$: something in the sample space certainly happens.
3. Additivity. If $A_1, A_2, A_3, \dots$ are mutually exclusive events, then

$$P(A_1 \cup A_2 \cup A_3 \cup \cdots) = P(A_1) + P(A_2) + P(A_3) + \cdots.$$

The third axiom says that for events that cannot overlap, probabilities simply add. (It is stated for a whole sequence of events, not just two, so that it also works for infinite sample spaces.)

These axioms say nothing about which numbers to use for a particular coin or die; they only say how any sensible assignment must behave. Everything else follows from them.

Consequences of the axioms

The complement rule. An event and its complement are mutually exclusive and together make up $\Omega$. By axioms 2 and 3, $P(A) + P(A^c) = P(\Omega) = 1$, so

$$P(A^c) = 1 - P(A).$$

This is often the easiest route to an answer: the probability of “at least one six in four rolls” is $1$ minus the probability of “no sixes in four rolls”.

Two further facts follow immediately. The empty event has probability $P(\varnothing) = 1 - P(\Omega) = 0$. And since $P(A^c) \ge 0$, every probability satisfies $0 \le P(A) \le 1$.

The addition rule (inclusion–exclusion). When $A$ and $B$ can overlap, adding $P(A)$ and $P(B)$ counts the overlap $A \cap B$ twice. Subtracting it once fixes this:

$$P(A \cup B) = P(A) + P(B) - P(A \cap B).$$

If $A$ and $B$ are mutually exclusive, $P(A \cap B) = 0$ and we are back to axiom 3. For three events the same idea gives

$$\begin{aligned} P(A \cup B \cup C) = {} & P(A) + P(B) + P(C) \\ & - P(A \cap B) - P(A \cap C) - P(B \cap C) \\ & + P(A \cap B \cap C). \end{aligned}$$

Monotonicity. If every outcome in $A$ is also in $B$ (written $A \subseteq B$), then $P(A) \le P(B)$. A more specific event can never be more probable than a more general one.

Equally likely outcomes

When the sample space is finite and every outcome is equally likely, the axioms force a simple formula. If $\Omega$ has $N$ outcomes, each has probability $1/N$, and an event $A$ containing $|A|$ outcomes has probability

$$P(A) = \frac{|A|}{|\Omega|} = \frac{\text{number of outcomes in } A}{\text{total number of outcomes}}.$$

Probability then becomes a matter of counting.

Worked example

Roll one fair die. Let $A = \{2, 4, 6\}$ (“even”) and $B = \{5, 6\}$ (“at least 5”). Find the probability of “even or at least 5”.

Step 1: the individual probabilities. All six outcomes are equally likely, so

$$P(A) = \frac{3}{6}, \qquad P(B) = \frac{2}{6}.$$

Step 2: the overlap. Only the outcome 6 is in both events, so $A \cap B = \{6\}$ and $P(A \cap B) = 1/6$.

Step 3: the addition rule.

$$P(A \cup B) = \frac{3}{6} + \frac{2}{6} - \frac{1}{6} = \frac{4}{6} = \frac{2}{3}.$$

Check by counting. $A \cup B = \{2, 4, 5, 6\}$ has four of the six outcomes, so $P(A \cup B) = 4/6$. Simply adding $3/6 + 2/6 = 5/6$ would have been wrong, because the 6 would have been counted twice.

A complement question. The probability of “neither even nor at least 5” is $1 - 2/3 = 1/3$; these are the outcomes $\{1, 3\}$.

What does a probability mean?

The axioms tell us how probabilities behave, but not what they are. Two interpretations are in common use, and both obey the same axioms.

• Frequency interpretation. The probability of an event is the long-run proportion of times it happens when the experiment is repeated many times under the same conditions. Saying a fair coin lands heads with probability $1/2$ means that in a long sequence of tosses about half are heads. The law of large numbers makes this connection precise.
• Degree-of-belief (Bayesian) interpretation. A probability expresses how strongly someone believes a statement, given the information they have. This makes sense even for one-off events that cannot be repeated, such as “this patient has the disease” or “it will rain tomorrow”. Bayes’ theorem describes how such beliefs should change in the light of evidence.

Most of the mathematics is the same under either reading. The interpretations differ mainly in how statistical conclusions are phrased, a difference that becomes important in hypothesis testing and confidence intervals.

Common misunderstandings

“All outcomes are equally likely.” Only in special cases. The sum of two dice can be anything from 2 to 12, but a sum of 7 is six times as likely as a sum of 2. The counting formula $|A|/|\Omega|$ is valid only when the outcomes really are equally likely.

“P(A or B) = P(A) + P(B).” Only when $A$ and $B$ are mutually exclusive. Otherwise the overlap must be subtracted.

“After a run of tails, heads is due.” For a fair coin tossed independently, each toss still has probability $1/2$ of heads, whatever happened before. This mistake is known as the gambler’s fallacy. Long-run frequencies settle down because the early run is diluted by many later tosses, not because the coin compensates. See independence.

“Probability 0 means impossible.” In a finite sample space where every outcome has positive probability, yes. For continuous quantities, no: a waiting time of exactly 2 minutes is possible even though the probability of that single exact value is 0. Only intervals of values have positive probability; see random variables.

Further reading

• Joseph K. Blitzstein and Jessica Hwang, Introduction to Probability, 2nd ed., CRC Press, 2019. A clear, example-driven development of probability from the axioms.
• David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton, 2007. An intuitive, non-mathematical introduction to chance.

Related articles

• Conditional Probability — The probability of an event once we know that another event has occurred.
• Random Variables — A random variable assigns a number to each outcome of a random experiment; its distribution is described by a PMF, a PDF, or a CDF.
• Probability Distributions — A description of which values a random variable can take and how probability is spread over them.
• Introduction to Statistics — What statistics is for, the difference between describing data and drawing conclusions from it, and how this encyclopedia is organized.