1 Probabilities: the basics
1.1 Sample spaces and events#
We begin with a sample space \Omega, which is just a collection of all of the possible outcomes over which we are uncertain in the given situation.
For example, it could be the possible results of rolling a die, or coin flips, different ways the weather could go, or the decisions somebody else is going to make without your input.
A sample space is the set of all of these outcomes. By a “set”, I just mean a collection of other things. I’ll have to talk about these quite a lot, so you may want to go and take a look at the set theory primer now available on my website if you want to get more familiar with them.
A set of some of the outcomes is called an event. For example, an event could be the die rolling 2 or 3, or it could be the weather being rainy or overcast. An event can also contain just a single outcome, and in this case although an event is formally distinct from an outcome, we can treat them like they’re the same thing. (For example, the event that the die rolls 1).
An event can also be that any of the possible outcomes happens. In this case, it is equal to the sample space \Omega. This is why I called it the universal event last week. The sample space and the universal event are the same thing: the set of all of the things that could happen.
The set containing no outcomes at all is also an event, albeit one that can never happen. We denote this by \varnothing and we can call it the empty event.
1.2 Combining events#
If P and Q are events, we can write P \cup Q for the event that something in either P or Q (or both) happens. This is a set-theoretic union. But, if you like, you can also write P \vee Q. Technically this is an abuse of notation (more on that later), but we won’t worry about it.
Figure 1. The union P \cup Q: everything in P, in Q, or in both.
?Check your understanding: unions
- If event P is that the die rolls an even number, and event Q is that the die rolls an odd number, what is P \cup Q?
Show answer
The event that the die rolls any number at all. Every outcome is either even or odd, so P \cup Q is the whole sample space \Omega. - If P is that it will rain today, and Q is that it will rain hard today, what is P \cup Q?
Show answer
Just P: that it will rain today. Every way of raining hard is a way of raining, so Q is a subset of P, and adding it to P adds nothing. - If P is that my opponent’s hand contains the Queen of Spades, and Q is that my opponent’s hand contains the Jack, King or Ace of Spades, what is P \cup Q?
Show answer
The event that their hand contains the Jack, Queen, King or Ace of Spades. - If P is that it will rain today, and Q is that it will rain today or tomorrow, what is P \cup Q?
Show answer
Similarly, for events P and Q we can write P \cap Q for the event that something in both P and Q happens. This is a set-theoretic intersection. Similarly to before, you can write P \wedge Q or P \& Q instead if you like.
Figure 2. The intersection P \cap Q: everything in both P and Q.
?Check your understanding: intersections
- If P is that the die rolls an even number, and Q is that the die rolls an odd number, what is P \cap Q?
Show answer
The empty event \varnothing. No outcome is both even and odd, so there is nothing in both. - If P is that it will rain today, and Q is that it will rain hard today, what is P \cap Q?
Show answer
Just Q: that it will rain hard. Since Q is a subset of P, everything in both is just everything in Q. - If P is that the card drawn is a heart, and Q is that it is a King, what is P \cap Q?
- If P is that the die rolls an even number, and Q is that it rolls higher than 3, what is P \cap Q?
Show answer
Finally, any event P has a complement P^c, which we will often denote by \neg P. This is the event that contains all the outcomes which are in the sample space but not in P.
Figure 3. The complement P^c: everything in \Omega that is not in P.
?Check your understanding: complements
- If P is that the die rolls a 6, what is \neg P?
Show answer
- If P is that the die rolls an even number, what is \neg P?
Show answer
The event that it rolls an odd number. Notice that P \cup \neg P is the whole sample space and P \cap \neg P is empty, which is true of any event and its complement. - If P is that it will rain today, what is \neg P?
Show answer
The event that it will not rain today.
1.3 Set-theoretic vs logical operators#
In general, and officially, I take the sample space to be a set of outcomes, and events to be subsets (sets which might be smaller) of those outcomes. The proper way to combine these kinds of events to get other events is to use the set theoretic notions of union and intersection. The union of sets X and Y is the set that contains everything in either X or Y, and the intersection of X and Y is the set that contains everything in both X and Y. (For much more detail on this, see the primer.)
Other authors, including Schwarz in BDRC, prefer to talk about propositions, and to use the logical notions of conjunction (\wedge) and disjunction (\vee)—I just mean “and” and “or”.1
Technically, these are different. “And” and “or” are truth-functional conditions which take truth-apt objects like sentences or propositions and return further sentences or propositions whose truth values are grounded in the truth values of their component parts. “Intersection” and “union” are set-theoretic operators which combine two sets to form another set.
They do different things. But they are very tightly related and behave exactly the same way in most contexts, so in this course we will mostly ignore the differences between them. In other words, I’m not going to penalise you if you write \wedge instead of \cap.
1.4 On non-measurable sets#
Generally in mathematics a Probability Space comes with a sample space \Omega and a specification of events \Sigma—a list of all those subsets of \Omega which “count” as events. (It also has what’s called a probability measure, which we’re just about to talk about.)
The reason why the specification of events \Sigma has to exist in general is that in general, not every subset of a sample space can coherently be assigned a probability, if we want certain intuitively plausible regularities to hold. The reasons why are quite complicated. For our purposes, we can ignore the problem, and we can generally assume that any set of outcomes counts as an event.
OptionalWhy some sets cannot be given a probability
This material is definitely not examinable. It is here for mathematically-inclined students who want a fuller explanation, and it relies on a number of mathematical concepts you may not have come across before. My recommendation is to read on until you have finished the lecture notes, and to come back to this only if you are interested.
Suppose our sample space \Omega is the space \mathbb{R} mod \mathbb{Z}. If you have taken enough algebra, you will recognise this as the quotient group. I won’t explain exactly what this notation means, but essentially \Omega is just the real numbers but where anything that’s not between 0 and 1 has whole numbers added or subtracted until it is. (For example, 3.375 gets sent to 0.375.) You can also think of it as being the unit-circumference circle, because on such a circle going round a distance x and going a distance x + n leave you in exactly the same place.
Figure 4. Adding or subtracting whole numbers drags every real number into the shaded interval [0,1). Two families are shown: -0.625 and 1.375 both land on 0.375, and -0.3 and 1.7 both land on 0.7.
Figure 5. The same two points, on the circle of circumference 1. Distance is measured clockwise from 0, so going once all the way round returns you to where you started — which is why each point has infinitely many names, all differing by whole numbers.
We might want to assign probabilities to subsets of this space under some function f with the following properties. The first three are just the Kolmogorov axioms, with countable additivity replacing finite additivity:
- (i) Non-negativity. f(\sigma) \geq 0.
- (ii) Normalisation. f(\Omega) = 1.
- (iii) Countable additivity. If there are countably many \sigma_i and they are pairwise disjoint, then f\left(\bigcup_i \sigma_i\right) = \sum_i f(\sigma_i).
- (iv) Rotational invariance. If \sigma_2 = \sigma_1 + q = \{r : \exists s \in \sigma_1 \text{ s.t. } r = s + q\}, with q a rational number, then f(\sigma_1) = f(\sigma_2).
It turns out that there is no way to assign probabilities to all of the subsets of this space in a way that satisfies all four conditions. (At least, this can be shown if we assume, as most mathematicians do, a slightly controversial principle known as the Axiom of Choice.)
To show this, we partition the space as follows. We first consider \mathbb{R} mod \mathbb{Q}, on which we treat two real numbers the same not only if they differ by an integer, but also if they differ by any rational number. More precisely, it is a partition of the real numbers into subsets (in this case, “equivalence classes”), where each subset consists of all those numbers for which the difference between them is a rational number. For example, the set of rational numbers is one such subset, while the set of all numbers whose distance from \pi is a rational number would be another such subset. At any rate, there are (uncountably) infinitely many such subsets.
From each subset, we take one member of that subset—it doesn’t matter which one—and collect them all together to form a new set V, known as a Vitali set. (Note that this step is an application of the Axiom of Choice.) We then enumerate the rational numbers as q_1, q_2, q_3, \dots, and consider the collection of sets
It can be shown (I won’t do it here) that these subsets V + q_i partition \Omega, and also that they are rotations of each other. Therefore the probability of each set—or, rather, event—V + q_i is the same.
Figure 6. Adding a rational q_i to every point of V turns the whole pattern rigidly about the circle. The copies are congruent, so rotational invariance forces them all to have the same probability; and every point of \Omega lies in exactly one of them, so together they partition \Omega. Those two facts are what the argument then plays off against each other. Note that V contains vastly more points than are drawn here: it is both dense and uncountable in the circle. The handful of dots is only there to make it visible that every point is being rotated at once.
If the probability of each V + q_i is zero, then by countable additivity f(\Omega) = \sum_i f(V + q_i) = 0 + 0 + \dots = 0, contradicting normalisation.
If instead the probability of each V + q_i is some \varepsilon > 0, then for some n we have n\varepsilon > 1; and since (V + q_1) \cup \dots \cup (V + q_n) is a subset of \Omega made of n disjoint pieces each of probability \varepsilon, we get f\left((V + q_1) \cup \dots \cup (V + q_n)\right) = n\varepsilon > 1, contradicting the combination of non-negativity, normalisation and countable additivity.
What this shows is that certain conditions we might like to impose on a probability measure prohibit us from assigning probabilities to certain strange subsets like the V + q_i sets. As far as probabilities go, that could mean we should reject principles like rotational invariance, or it could mean that the probabilities of certain things that look like they should be events are not well-defined.
1.5 Mutual exclusivity, joint exhaustiveness#
Sometimes, two events have no outcome in common at all. When events X and Y are like this, we say that they are mutually exclusive or disjoint, and the condition for it is that X \cap Y = \varnothing, the empty event.
Figure 7. Mutually exclusive (disjoint) events: X \cap Y = \varnothing.
Sometimes, two or more events are such that it is guaranteed that at least one will happen. In this case, we say that the events are jointly exhaustive, and the condition for it is that X_1 \cup \dots \cup X_n = \Omega, i.e., every outcome is in at least one of the events in question.
And when a set of events are mutually exclusive and jointly exhaustive, we say that they partition the sample space.
Figure 8. A partition of \Omega: the cells are pairwise disjoint and jointly exhaustive, so exactly one of them obtains.
Remember from last week: we want to also require that the set of states of nature is mutually exclusive and jointly exhaustive. So, the states of nature partition the sample space for each act.
1.6 Probabilistic independence#
1.7 Probabilistic independence is not mutual exclusivity#
These two are easy to run together, but they are actually close to opposites.
Suppose A and B are mutually exclusive, and both have positive credence. Then A \cap B = \varnothing, so \Cr(A \cap B) = 0, while \Cr(A) \cdot \Cr(B) > 0. So they are not probabilistically independent.
In fact they are as dependent as two events can be: learning that A happened takes your credence in B straight to zero.
Figure 9. Mutually exclusive: A and B cannot both happen, so learning that A happened rules B out entirely. This is the opposite of probabilistic independence, not an instance of it.
Figure 10. Probabilistically independent: B takes up the same fraction of A as it does of \Omega. The shaded cell is A \cap B.
- Strictly speaking, since Schwarz characterises propositions as sets of possible worlds, these actually amount to exactly the same thing as the set-theoretic notions of intersection and union anyway. ↩