Bayes' rule

Definition

Bayes' rule is a mathematical formula used to find a conditional probability by reversing the direction of known information.

$P(A|B)=\frac{P(B|A)\,P(A)}{P(B)}$

Where:

$P(A|B)$ = probability of event $A$ given that event $B$ has occurred
$P(B|A)$ = probability of event $B$ given that event $A$ has occurred
$P(A)$ = prior probability of $A$
$P(B)$ = total probability of $B$

A more complete version is:

$P(A|B)=\frac{P(B|A)\,P(A)}{P(B|A)P(A)+P(B|\neg A)P(\neg A)}$

This is especially useful when event $B$ can happen for more than one reason.

Main Content

1. Conditional Probability and Reversing Information

Conditional probability means the probability of one event happening given another event has already happened. For example, $P(\text{rain}|\text{cloudy})$ means the chance of rain if the sky is cloudy.
Bayes' rule is important because it allows us to reverse the conditioning. Often we know $P(B|A)$ more easily than $P(A|B)$ . Bayes' rule helps convert one into the other.

To understand why this matters, suppose:

$A$ = a person has a disease
$B$ = a test comes back positive

Doctors often know:

how likely a positive test is if someone is sick, $P(B|A)$ , but they really want:
how likely someone is sick if the test is positive, $P(A|B)$

These two probabilities are not the same. Bayes' rule gives the correct relationship between them.

A useful way to think about it is:

$\text{Posterior} \propto \text{Likelihood} \times \text{Prior}$

This means the new probability depends on:

the earlier belief,
the strength of the evidence,
and the need to normalize by the total probability of the evidence.

2. Prior, Likelihood, and Posterior

Prior probability

is the belief about an event before observing new data. It represents background knowledge or initial expectation. For example, if only 1 out of 1000 people have a rare disease, then that 0.1% is the prior.

Likelihood

measures how likely the observed evidence is under a given hypothesis. In a medical test, if a sick person has a 99% chance of testing positive, then the likelihood is very high for the disease hypothesis.

These two combine to form the posterior probability, which is the updated belief after considering the evidence.

Bayes' rule can be remembered as:

Prior

: what we already think

Evidence

: what we observe

Posterior

: what we conclude after updating

Example: If a disease is very rare, even a highly accurate test may produce many false positives compared with true positives. So a positive result does not automatically mean the disease is likely. Bayes' rule balances:

the rarity of the disease,
the accuracy of the test,
and the meaning of a positive result.

This is why Bayes' rule is often surprising to people: it shows that base rates matter. A test can be accurate and still not guarantee a high probability of disease if the condition itself is uncommon.

3. Total Probability and Real-Life Interpretation

The denominator in Bayes' rule, $P(B)$ , is the total probability of the observed evidence. It ensures that the final probability is properly scaled and lies between 0 and 1.
Often, $P(B)$ is found using the law of total probability, especially when evidence can arise from multiple causes.

If event $B$ can happen either because $A$ happened or because $A$ did not happen, then:

$P(B)=P(B|A)P(A)+P(B|\neg A)P(\neg A)$

This means we add up all possible ways the evidence can occur.

Example: Medical Test

Suppose:

1% of people have a disease: $P(D)=0.01$
Test sensitivity: $P(+|D)=0.99$
False positive rate: $P(+|\neg D)=0.05$

We want $P(D|+)$ , the probability of disease given a positive test.

Using Bayes' rule:

$P(D|+)=\frac{P(+|D)P(D)}{P(+|D)P(D)+P(+|\neg D)P(\neg D)}$

$P(D|+)=\frac{0.99 \times 0.01}{(0.99 \times 0.01)+(0.05 \times 0.99)}$

$P(D|+)=\frac{0.0099}{0.0099+0.0495}=\frac{0.0099}{0.0594}\approx 0.1667$

So even with a positive result, the probability of actually having the disease is only about 16.67%.

This shows why Bayes' rule is essential in real situations: it prevents overconfidence based only on test results.

Visual Intuition

Prior belief + new evidence -> Updated belief

Rare disease
   |
   v
Positive test
   |
   v
Not always "likely disease"
because false positives also matter

Bayes' rule is not just a formula; it is a framework for thinking logically under uncertainty.

Working / Process

1. Identify the hypothesis and evidence

Decide what event you want to find the probability of.
Let that event be $A$ , and the observed evidence be $B$ .
Example: $A$ = person has disease, $B$ = test is positive.

2. Collect the probabilities needed

Find the prior probability $P(A)$ .
Find the conditional probability $P(B|A)$ .
Find the probability of evidence under the alternative, $P(B|\neg A)$ , if needed.
Use the law of total probability to calculate $P(B)$ .

3. Apply Bayes' rule and interpret the result

Substitute values into the formula: $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$
Simplify carefully.
Interpret the answer as an updated probability, not just a mechanical result.
Check whether the result makes sense in context.

Step-by-step example

Suppose:

$P(A)=0.2$
$P(B|A)=0.8$
$P(B|\neg A)=0.3$

First calculate: $P(B)=0.8(0.2)+0.3(0.8)=0.16+0.24=0.40$

Then: $P(A|B)=\frac{0.8(0.2)}{0.40}=\frac{0.16}{0.40}=0.4$

So the updated probability of $A$ given $B$ is 0.4.

This process shows that evidence can increase or decrease belief depending on how strongly it supports the hypothesis compared with other possible explanations.

Advantages / Applications

Improves decision-making under uncertainty

Bayes' rule helps make rational choices when information is incomplete. It is used in risk assessment, forecasting, and planning.

Widely used in medical diagnosis and testing

It helps doctors interpret test results correctly by accounting for disease prevalence and test accuracy.

Foundation of modern machine learning and AI

Bayesian methods are used in spam filtering, classification, prediction, and probabilistic reasoning.

Useful in everyday reasoning

It helps people think more carefully about evidence, avoiding mistakes caused by ignoring base rates.

Supports updating beliefs as new data arrives

This is valuable in science, finance, weather prediction, and experimental analysis.

Helps compare competing explanations

When multiple causes can explain the same evidence, Bayes' rule identifies which is more plausible.

Summary

Bayes' rule updates probability using new evidence.
It connects prior belief, likelihood, and posterior probability.
It is especially useful when interpreting evidence in uncertain situations.
Important terms to remember
Prior
Likelihood
Posterior
Conditional probability
Total probability