3 Conditional probabilities and Conditionalisation

3.1 Defining conditional probabilities#
Definition (Conditional probability)
\Cr(A \mid B) = \frac{\Cr(A \cap B)}{\Cr(B)} \qquad \text{provided } \Cr(B) > 0.

In these diagrams areas represent probabilities, and in general probabilities behave very much like areas. When we suppose B, the new area lying in A is A \cap B; but the total area over which we are calculating is now the area of B, not the area of \Omega, which was 1. That is why, to find how likely A is on the condition that B, we take its new area, A \cap B, and divide it by the new total area, B.

Figure 11. Before conditioning: \Omega is everything, and the shaded region A \cap B is measured against all of it.
A B Ω
1 / 2

Step through the two frames: the shaded region is the same in both. What changes is what it is being measured against.

So the formula does exactly that: \Cr(A \cap B) measures the overlap, and dividing by \Cr(B) rescales it so that B, rather than \Omega, counts as the whole. It is also why we require \Cr(B) > 0: if B has no probability, there is nothing to rescale against.

This also gives us a second, equivalent way of saying that two events are probabilistically independent, which we met back in section 1. Where \Cr(B) > 0,

\Cr(A \cap B) = \Cr(A) \cdot \Cr(B) \qquad \text{is the same as} \qquad \Cr(A \mid B) = \Cr(A).
The second form is arguably the more intuitive one, since it says directly that learning whether B happened makes no difference at all to how likely A is.

3.2 Conditionalisation#
Principle (Conditionalisation)
Suppose your credences are given by \Cr, and then you learn E and nothing more. Your new credence in any event A should be your old credence in A conditional on E:
\Cr_{\text{new}}(A) = \Cr(A \mid E).
3.3 A visual explanation of conditionalisation#
Figure 13. Stage 1. The sample space \Omega, with areas representing probabilities. B is the event we are going to condition on, and A lies wholly inside it.
C D A B Ω
1 / 5
3.4 The multiplication rule and total probability#

Sometimes it is useful to use conditional probabilities to calculate the unconditional probability of an event. Suppose that the sample space \Omega can be partitioned into B_1, B_2, \dots, B_n. Then the law of total probability states that

\Cr(A) = \Cr(B_1)\Cr(A \mid B_1) + \Cr(B_2)\Cr(A \mid B_2) + \dots + \Cr(B_n)\Cr(A \mid B_n).

Why is this true? Note that, since the B_i partition \Omega,

\Cr(A) = \Cr(A \cap B_1) + \Cr(A \cap B_2) + \dots + \Cr(A \cap B_n).
Then try substituting the definition of conditional probability into the total probability formula.

Exercise
Prove the law of total probability.
Figure 18. The law of total probability. The B_i partition \Omega, so A is cut into the disjoint pieces A \cap B_i, and their probabilities sum to \Cr(A).
B1 B2 B3 B4 A Ω
3.5 Bayes’ theorem#

What we have, straight from the definition, is a pair of equations:

\Cr(A \mid B) = \frac{\Cr(A \cap B)}{\Cr(B)} \qquad\text{and}\qquad \Cr(B \mid A) = \frac{\Cr(A \cap B)}{\Cr(A)}.

What we want is a way of getting from one conditional probability to the other:

Principle (Bayes’ theorem)
\Cr(A \mid B) = \frac{\Cr(B \mid A)\,\Cr(A)}{\Cr(B)} \qquad \text{provided } \Cr(A), \Cr(B) > 0.

Sometimes you know one of these conditional probabilities and want the other, and Bayes’ theorem is what gets you across.

Exercise
Prove Bayes’ theorem.
3.6 Bayes factors#

Suppose you are weighing two hypotheses, H_1 and H_2, and some evidence E arrives. The Bayes factor is

\frac{\Cr(E \mid H_1)}{\Cr(E \mid H_2)}.

It measures how much better E is explained by H_1 than by H_2, and it is the whole of what the evidence contributes. That is easiest to see in odds form:

\underbrace{\frac{\Cr(H_1 \mid E)}{\Cr(H_2 \mid E)}}_{\text{posterior odds}} = \underbrace{\frac{\Cr(E \mid H_1)}{\Cr(E \mid H_2)}}_{\text{Bayes factor}} \times \underbrace{\frac{\Cr(H_1)}{\Cr(H_2)}}_{\text{prior odds}}

So the Bayes factor is exactly the multiplier: it says how far the evidence moves you, and your prior says where you started from. A Bayes factor of 1 means the evidence is equally expected either way, and moves you not at all.

Example
Suppose you consider it equally likely that a coin is biased so that it always lands heads (H_1) or is fair (H_2), and you flip it and get heads (E). The probability of its landing heads given that it was biased is 1, and given that it was fair is \nicefrac{1}{2}, so the Bayes factor is 1 / 0.5 = 2. The odds between biased and fair therefore move from 1:1 to 2:1 — in other words, a \nicefrac{2}{3} chance that the coin is biased and a \nicefrac{1}{3} chance that it is fair.