Nexus Learn

Lecture 20: Probabilistic Reasoning and Bayes' Rule

Lecture Notes CS-511 Artificial Intelligence 6 min read Updated 8 Nov 2026

↓ Download notes (Word)

On this page
  1. Learning Outcomes
  2. 1. Probability Basics
  3. 2. Bayes' Rule
  4. 3. Naive Bayes
  5. 4. Common Mistakes
  6. Word Bank
  7. Self-Check Questions

Learning Outcomes

SLO

After this lecture you will be able to...

CLO

Bloom

SLO 20.1

Compute joint, marginal and conditional probabilities from a table and test independence

CLO3

C3 Apply

SLO 20.2

Apply Bayes' rule to update a belief from evidence and explain the effect of the prior

CLO3

C3 Apply

SLO 20.3

Apply a Naive Bayes classifier to a small example and explain its independence assumption

CLO3

C3 Apply

Note: all medical and SMS data in this lecture are made up for teaching; they are realistic, not official figures.

1. Probability Basics

§1.1 Why AI needs probability

Slide L20-S06

The world is uncertain: causes are hidden (a doctor sees fever, not the virus), sensors are noisy, and rules have too many exceptions to list. Probability gives a consistent number from 0 (impossible) to 1 (certain). P(not A) = 1 - P(A). A random variable takes values with probabilities, e.g. Weather in {sunny, rain}.

§1.2 Joint, marginal and conditional probability

Slide L20-S07

fever no fever total

dengue 45 5 50

no dengue 190 760 950

total 235 765 1000

P(dengue) = 0.050 P(fever) = 0.235

P(dengue AND fever) = 0.045

P(dengue | fever) = 0.045 / 0.235 = 0.191

P(fever | dengue) = 0.045 / 0.050 = 0.900

P(dengue) x P(fever) = 0.0118, not 0.045

Independent? no

Kind

Meaning

Formula

Value

Joint

Both together

P(D and F)

0.045

Marginal

One variable alone (row/column total)

P(D)

0.05

Conditional

A when B is known

P(D | F) = P(D and F) / P(F)

0.191

Careful: P(fever | dengue) = 0.90 but P(dengue | fever) = 0.19 - the order matters.

§1.3 Rules and independence

Slide L20-S08

Rule

Formula

Product rule

P(A and B) = P(A | B) P(B) = P(B | A) P(A)

Sum rule (marginalisation)

P(A) = P(A and B) + P(A and not B)

Independence

P(A | B) = P(A), i.e. P(A and B) = P(A) P(B)

2. Bayes' Rule

§2.1 The formula

Slide L20-S10

P(H | E) = P(E | H) x P(H) / P(E)

P(E) = P(E | H) P(H) + P(E | not H) P(not H)

Name

Symbol

Meaning

Prior

P(H)

Belief before the evidence

Likelihood

P(E | H)

How often the evidence appears if H is true

Evidence

P(E)

How often the evidence appears overall

Posterior

P(H | E)

Belief after the evidence

§2.2 The dengue test

Slide L20-S11

Prior P(D) = 0.02

Sensitivity P(+|D) = 0.9

False positive P(+|not D) = 0.05

P(+) = 0.9 x 0.02 + 0.05 x 0.98

= 0.067

P(D|+) = 0.9 x 0.02 / 0.067 = 0.269

Second positive test: 0.269 -> 0.869

Effect of the prior (same test):

prior 0.001 -> P(D|+) = 0.018

prior 0.02 -> P(D|+) = 0.269

prior 0.2 -> P(D|+) = 0.818

prior 0.5 -> P(D|+) = 0.947

Frequency view: of 1,000 people, 20 have dengue (18 test positive) and 980 are healthy (49 test positive). 67 positives, 18 really ill: 18 / 67 = 0.27.

§2.3 The prior and repeated tests

Slide L20-S12

The same test means very different things in different groups: P(D | +) is 0.018 in a random survey (prior 0.001) but 0.947 in an outbreak ward (prior 0.5). Ignoring the prior is the base-rate fallacy. Bayes is sequential: the posterior becomes the next prior (0.02 -> 0.269 -> 0.869 after two positive tests).

3. Naive Bayes

§3.1 The classifier

Slide L20-S14

score(C) = P(C) x P(e1 | C) x ... x P(en | C); choose the class with the larger score. The naive assumption: the clues are independent once the class is known. Laplace smoothing: P(w | C) = (count + 1) / (words in C + vocabulary size), so an unseen word never gives probability 0.

§3.2 Worked example: SMS spam filter

Slide L20-S15

spam = ["win free prize now", "free mobile package offer", "claim your prize now", "free recharge win"]

ham = ["class at nine tomorrow", "send me the notes", "free period after lunch", "see you at the bus stop",

"prize distribution ceremony tomorrow", "call me now"]

msg = "free prize now"

vocab = sorted(set(" ".join(spam + ham).split()))

cs, ch = Counter(" ".join(spam).split()), Counter(" ".join(ham).split())

ns, nh, V = sum(cs.values()), sum(ch.values()), len(vocab)

prior_s, prior_h = len(spam) / 10, len(ham) / 10

print(f"Training: {len(spam)} spam, {len(ham)} ham; vocabulary {V} words; spam words {ns}, ham words {nh}")

ps, ph = prior_s, prior_h

for w in msg.split():

a, b = (cs[w] + 1) / (ns + V), (ch[w] + 1) / (nh + V)

print(f" P('{w}'|spam) = ({cs[w]}+1)/({ns}+{V}) = {a:.4f} P('{w}'|ham) = ({ch[w]}+1)/({nh}+{V}) = {b:.4f}")

ps *= a; ph *= b

print(f"score spam = {ps:.3e} score ham = {ph:.3e}")

print(f"P(spam | '{msg}') = {ps / (ps + ph):.3f}")

Training: 4 spam, 6 ham; vocabulary 28 words; spam words 15, ham words 25

P('free'|spam) = (3+1)/(15+28) = 0.0930 P('free'|ham) = (1+1)/(25+28) = 0.0377

P('prize'|spam) = (2+1)/(15+28) = 0.0698 P('prize'|ham) = (1+1)/(25+28) = 0.0377

P('now'|spam) = (2+1)/(15+28) = 0.0698 P('now'|ham) = (1+1)/(25+28) = 0.0377

score spam = 1.811e-04 score ham = 3.224e-05

P(spam | 'free prize now') = 0.849

scikit-learn MultinomialNB: P(spam) = 0.849

'see you after class': P(spam) = 0.088

'win a free recharge': P(spam) = 0.937

§3.3 Probability vs certainty factors

Slide L20-S16

Certainty factors (L19)

Probability

Scale

-1 to +1

0 to 1

Combining

a + b(1 - a), heuristic

Exact rules

Numbers from

Expert judgement

Data or experts

Problem

Can be inconsistent

Needs many numbers -> Bayesian networks (L21)

4. Common Mistakes

Wrong idea

Correct idea

P(A | B) = P(B | A).

0.19 vs 0.90 in the clinic.

A positive 90% test means 90% ill.

Depends on the prior.

P(A and B) = P(A)P(B) always.

Only if independent.

Naive Bayes believes words are independent.

It assumes it to simplify.

An unseen word gets probability 0.

Use Laplace smoothing.

Word Bank

Word

Simple meaning

Example

Probability

Number from 0 to 1 for how likely

P(rain) = 0.3

Conditional probability

Probability when something is known

P(dengue | fever)

Independent

One does not change the other

Two coin tosses

Prior

Belief before evidence

2% have dengue

Posterior

Belief after evidence

27% after a + test

Base-rate fallacy

Forgetting the prior

Thinking + test = 90% ill

Laplace smoothing

Add 1 to every count

(0+1)/(15+28)

Self-Check Questions

  1. From the clinic table find P(fever | no dengue). (see §1.2)
  2. State the product and sum rules. (see §1.3)
  3. Name the four parts of Bayes' rule. (see §2.1)
  4. Repeat the dengue calculation with a prior of 0.1. (see §2.2)
  5. Why does the prior change the answer so much? (see §2.3)
  6. Why is Laplace smoothing needed? (see §3.1)

Reading: Russell & Norvig (4e) 12.1-12.6 (quantifying uncertainty, Bayes' rule, naive Bayes).

← All CS-511 material