Nexus Learn

Lecture 25: Decision Trees

Lecture Notes CS-511 Artificial Intelligence 6 min read Updated 8 Nov 2026

↓ Download notes (Word)

On this page
  1. Learning Outcomes
  2. 1. What Is a Decision Tree?
  3. 2. Choosing the Best Question
  4. 3. Decision Trees in Python
  5. 4. Common Mistakes
  6. Word Bank
  7. Self-Check Questions

Learning Outcomes

SLO

After this lecture you will be able to...

CLO

Bloom

SLO 25.1

Explain the parts of a decision tree and use a tree to classify a new example

CLO3

C2 Understand

SLO 25.2

Calculate entropy and information gain to choose the best splitting attribute (ID3)

CLO3

C3 Apply

SLO 25.3

Build a decision tree with scikit-learn and control overfitting with the tree depth

CLO3

C3 Apply

1. What Is a Decision Tree?

§1.1 A tree of questions

Slide L25-S06

A decision tree is a supervised learning model that classifies an example by asking a chain of questions about its features. It is learned from labelled examples. Running example: 14 Sundays of a friendly cricket match in Burewala - was it played?

Day

Outlook

Temperature

Humidity

Wind

Played?

1

sunny

hot

high

weak

no

2

sunny

hot

high

strong

no

3

overcast

hot

high

weak

yes

4

rain

mild

high

weak

yes

5

rain

cool

normal

weak

yes

6

rain

cool

normal

strong

no

7

overcast

cool

normal

strong

yes

8

sunny

mild

high

weak

no

9

sunny

cool

normal

weak

yes

10

rain

mild

normal

weak

yes

11

sunny

mild

normal

strong

yes

12

overcast

mild

high

strong

yes

13

overcast

hot

normal

weak

yes

14

rain

mild

high

strong

no

Part

Meaning

In the cricket tree

Root node

The first question

Outlook?

Internal node

A later question

Humidity?, Wind?

Branch

One answer to a question

sunny / overcast / rain

Leaf

The final answer (class)

yes / no

To classify a new day, start at the root and follow the matching branches to a leaf. Rain, hot, high humidity, strong wind: Outlook? rain -> Wind? strong -> no. Temperature is not used: the tree keeps only useful questions.

§1.2 Trees are rules

Slide L25-S07

Every root-to-leaf path is one IF-THEN rule, like the expert systems of Lecture 19 - but learned from data, not written by an expert. Uses: bank loans, hospital triage, mango export grading, predicting which mobile customers will leave.

IF outlook = overcast THEN play

IF outlook = sunny AND humidity = high THEN no play

IF outlook = sunny AND humidity = normal THEN play

IF outlook = rain AND wind = strong THEN no play

IF outlook = rain AND wind = weak THEN play

Two families: ID3 (Ross Quinlan, 1986) chooses questions with entropy; CART (Breiman and others, 1984) uses the Gini index and is the method in scikit-learn.

2. Choosing the Best Question

§2.1 Entropy

Slide L25-S09

Entropy measures how mixed (uncertain) a group is, in bits: H(S) = - Σ p_i log2 p_i. A pure group has entropy 0; a 50/50 group of two classes has entropy 1.

14 yes, 0 no entropy = 0.000 bits

7 yes, 7 no entropy = 1.000 bits

9 yes, 5 no entropy = 0.940 bits

3 yes, 1 no entropy = 0.811 bits

Gini of 9 yes, 5 no = 0.459

9 yes, 5 no: p(yes) = 9/14 = 0.643, p(no) = 5/14 = 0.357

H = -0.643 x log2(0.643) - 0.357 x log2(0.357) = 0.410 + 0.530 = 0.940 bits

Gini index (CART): 1 - Σ p_i². For 9 yes, 5 no it is 0.459; also 0 when pure.

§2.2 Information gain

Slide L25-S10

Gain(S, A) = H(S) - Σ over values v of A ( |S_v| / |S| ) x H(S_v)

Root: 9 yes, 5 no -> entropy 0.940

outlook = sunny 2 yes 3 no entropy 0.971

outlook = overcast 4 yes 0 no entropy 0.000

outlook = rain 3 yes 2 no entropy 0.971

Gain(outlook ) = 0.247

Gain(temperature) = 0.029

Gain(humidity ) = 0.152

Gain(wind ) = 0.048

By hand: Gain(outlook) = 0.940 - (5/14 x 0.971 + 4/14 x 0 + 5/14 x 0.971) = 0.940 - 0.694 = 0.247. Outlook has the highest gain, so it is the root - 'overcast' alone gives a pure group of 4 days.

Exam tip: use the WEIGHTED average of the branch entropies - each branch is weighted by its share of the examples.

§2.3 Growing the whole tree: ID3

Slide L25-S11

ID3(examples, attributes):

if all examples have the same class: return a leaf

if no attributes left: return a majority-class leaf

A = attribute with the highest information gain

for each value v of A:

add branch A = v -> ID3(examples with A = v, attributes - A)

Gains inside outlook = sunny (5 days):

temperature 0.571, humidity 0.971, wind 0.020

outlook = overcast: yes

outlook = rain:

wind = strong: no

wind = weak: yes

outlook = sunny:

humidity = high: no

humidity = normal: yes

ID3 is recursive (it repeats itself on smaller groups) and greedy (Lecture 13): it chooses the best question now and never goes back.

3. Decision Trees in Python

§3.1 A tree for the 12 students

Slide L25-S13

For number features the tree asks 'feature <= threshold?', trying thresholds halfway between sorted values. Trees need no feature scaling, because each question looks at one feature only.

from sklearn.tree import DecisionTreeClassifier, export_text

m = DecisionTreeClassifier(criterion="entropy", random_state=0).fit(X, y)

print(export_text(m, feature_names=["hours", "attendance"]))

|--- attendance <= 72.50

| |--- class: fail

|--- attendance > 72.50

| |--- class: pass

new student 8 h, 60 % -> fail

new student 4 h, 85 % -> pass

new student 11 h, 60 % -> fail

§3.2 Overfitting and pruning

Slide L25-S14

300 loan applicants (monthly income and existing debt, thousand Rs), 15 % of the labels noisy, 70 % for training and 30 % for testing:

depth leaves train test

1 2 0.74 0.79

2 4 0.80 0.74

3 8 0.86 0.83

5 24 0.92 0.76

8 41 0.98 0.76

None 50 1.00 0.74

Depth 3 gives the best test accuracy (0.83). With no limit the tree reaches 100 % on the training data but only 0.74 on the test data - it has memorised the noise (overfitting). Pruning (cutting back branches) keeps the tree small: max_depth, min_samples_leaf or ccp_alpha.

Strengths

Weaknesses

Easy to read and explain

Overfits if grown too deep

No feature scaling

Unstable: small data change, new tree

Fast; handles categories

One feature per question (boxy boundaries)

4. Common Mistakes

Wrong idea

Correct idea

High entropy is good.

We want low entropy (pure groups) after a split.

Gain = entropy of the biggest branch.

Use the weighted average of all branches.

The deepest tree is the best.

Deep trees overfit; check test accuracy.

Every attribute must appear.

Useless attributes are left out.

ID3 finds the best tree overall.

It is greedy: best question at each step.

Word Bank

Word

Simple meaning

Example

Decision tree

A model that asks questions to reach an answer

Outlook? -> Wind? -> no

Root / leaf

First question / final answer

Outlook? / yes

Entropy

How mixed a group is (0 = pure)

9 yes 5 no = 0.940

Information gain

Drop in entropy after a split

Gain(outlook) = 0.247

Gini index

Another impurity measure (CART)

0.459

Greedy

Takes the best step now, never goes back

ID3

Overfitting

Memorising the training data

train 1.00, test 0.74

Pruning

Cutting back the tree

max_depth = 3

Self-Check Questions

  1. Classify: sunny, mild, normal humidity, weak wind. (see §1.1)
  2. Write the tree's rule for rain days. (see §1.2)
  3. Entropy of 2 yes, 6 no? (see §2.1)
  4. Wind splits the 14 days into weak (6 yes, 2 no) and strong (3 yes, 3 no). Gain(wind)? (see §2.2)
  5. Why is ID3 called greedy? (see §2.3)
  6. Why does a tree not need feature scaling? (see §3.1)
  7. Which depth would you choose from the overfit table, and why? (see §3.2)

Reading: Govers Ch 8; Russell & Norvig Ch 19.3 (learning decision trees); scikit-learn user guide 'Decision Trees'.

← All CS-511 material