Recap of Lecture 24 and Homework Check
LAST LECTURE · 5 MIN
Today: k-NN answers but cannot explain WHY. Can a model give its reasons, like a doctor asking question after question?
Today's Plan (50 minutes)
ROADMAP
0-5
Recap of Lecture 24 and homework check
S02
5-8
Plan and learning outcomes
S03-S04
8-15
Module 1: What is a decision tree?
S05-S07
15-27
Module 2: Choosing the best question - entropy and information gain
S08-S11
27-35
Module 3: Trees in Python, overfitting and pruning
S12-S14
35-43
Class activity: entropy, gain and reading a tree
S15
43-48
Common mistakes and recap quiz
S16-S17
48-50
Homework and next lecture
S18
Learning Outcomes (SLOs)
BY THE END OF THIS LECTURE YOU WILL BE ABLE TO
SLO 25.1
Explain the parts of a decision tree and use a tree to classify a new example
→ CLO3 · C2 Understand
SLO 25.2
Calculate entropy and information gain to choose the best splitting attribute (ID3)
→ CLO3 · C3 Apply
SLO 25.3
Build a decision tree with scikit-learn and control overfitting with the tree depth
→ CLO3 · C3 Apply
- Course Learning Outcome(s) served
- CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.
Slide 5
MODULE 1
What Is a Decision Tree?
Twenty questions, learned from data
A Tree of Questions
MODULE 1 · PARTS OF A TREE
CLASSIFYING A NEW DAY
Not used: temperature - the tree keeps only useful questions.
- New day: rain, hot, high humidity, strong wind
- Outlook? rain -> Wind? strong -> no (match off)
Trees Are Everywhere - and They Are Rules
MODULE 1 · WHY TREES ARE POPULAR
Bank loan
Hospital triage
Mango export
Job? -> own house? -> approve / reject
Fever? -> breathing problem? -> emergency / wait
Colour? -> size? -> grade A / B
THE TREE AS FIVE IF-THEN RULES
- IF outlook = overcast THEN play
- IF outlook = sunny AND humidity = high THEN no play
- IF outlook = sunny AND humidity = normal THEN play
- IF outlook = rain AND wind = strong THEN no play
- IF outlook = rain AND wind = weak THEN play
One path = one rule. Each root-to-leaf path is an IF-THEN rule.
Two families: ID3 (Quinlan, 1986) uses entropy; CART (Breiman, 1984) uses Gini - scikit-learn uses CART.
Slide 8
MODULE 2
Choosing the Best Question
Entropy and information gain
Entropy: How Mixed Is a Group?
MODULE 2 · ENTROPY
ENTROPY (SHANNON, 1948)
REAL OUTPUT: L25/ENTROPY_DEMO.PY
H(S) = - Σ p_i · log2(p_i)
- 14 yes, 0 no entropy = 0.000 bits
- 7 yes, 7 no entropy = 1.000 bits
- 9 yes, 5 no entropy = 0.940 bits
- 3 yes, 1 no entropy = 0.811 bits
- Gini of 9 yes, 5 no = 0.459
BY HAND
- 9 yes, 5 no: p(yes) = 9/14 = 0.643
- p(no) = 5/14 = 0.357
- H = -0.643·log2(0.643) - 0.357·log2(0.357)
- = 0.410 + 0.530 = 0.940 bits
Goal of a split: make the groups below as PURE as possible - entropy as LOW as possible.
Gini index (CART): 1 - Σ p_i² = 0.459 for 9 yes, 5 no. Also 0 when pure.
Information Gain: Which Question First?
MODULE 2 · INFORMATION GAIN
INFORMATION GAIN
Gain(S, A) = H(S) - Σ (|S_v| / |S|) · H(S_v)
REAL OUTPUT: L25/GAINS.PY
BY HAND
- Root: 9 yes, 5 no -> entropy 0.940
- outlook = sunny 2 yes 3 no entropy 0.971
- outlook = overcast 4 yes 0 no entropy 0.000
- outlook = rain 3 yes 2 no entropy 0.971
- Gain(outlook ) = 0.247
- Gain(temperature) = 0.029
- Gain(humidity ) = 0.152
- Gain(wind ) = 0.048
- Gain(outlook) = 0.940 - ( 5/14·0.971
- + 4/14·0.000
- + 5/14·0.971 )
- = 0.940 - 0.694 = 0.247
Winner: Outlook (0.247) - it becomes the root.
Why outlook? 'overcast' alone gives a pure group: 4 days, all played.
Growing the Whole Tree (ID3)
MODULE 2 · THE ALGORITHM
ID3 (QUINLAN, 1986)
REAL OUTPUT: L25/GROW.PY
- ID3(examples, attributes):
- if all examples have the same class: return a leaf
- if no attributes left: return a majority-class leaf
- A = attribute with the highest information gain
- for each value v of A:
- add branch A = v -> ID3(examples with A = v,
- attributes - A)
- Gains inside outlook = sunny (5 days):
- temperature 0.571, humidity 0.971, wind 0.020
- outlook = overcast: yes
- outlook = rain:
- wind = strong: no
- wind = weak: yes
- outlook = sunny:
- humidity = high: no
- humidity = normal: yes
Pure node
No attributes left
Depth limit
All one class - make a leaf.
Leaf with the majority class.
Stop early to avoid overfitting.
ID3 is greedy (L13): it takes the best question now and never goes back.
Slide 12
MODULE 3
Decision Trees in Python
scikit-learn, numbers and overfitting
A Tree for the 12 Students
MODULE 3 · SCIKIT-LEARN
CODE: L25/STUDENTS_TREE.PY
REAL OUTPUT
- from sklearn.tree import DecisionTreeClassifier, export_text
- m = DecisionTreeClassifier(criterion="entropy", random_state=0)
- m.fit(X, y)
- print(export_text(m, feature_names=["hours", "attendance"]))
- m.predict([[8, 60]])
- |--- attendance <= 72.50
- | |--- class: fail
- |--- attendance > 72.50
- | |--- class: pass
- new student 8 h, 60 % -> fail
- new student 4 h, 85 % -> pass
- new student 11 h, 60 % -> fail
- Numbers: the tree asks 'attendance <= 72.5?'
- One question splits all 12 perfectly.
- No scaling needed - one feature per question.
Discuss: the tree ignored study hours completely. With only 12 students, a very simple rule fits - is it fair?
Overfitting and Pruning
MODULE 3 · HOW DEEP?
REAL OUTPUT: L25/OVERFIT.PY (300 LOAN APPLICANTS)
- Depth 3: test 0.83 - the best.
- No limit: train 1.00 but test 0.74.
- Overfitting: the deep tree memorised noisy cases.
- Pruning: max_depth, min_samples_leaf, ccp_alpha.
- depth leaves train test
- 1 2 0.74 0.79
- 2 4 0.80 0.74
- 3 8 0.86 0.83
- 5 24 0.92 0.76
- 8 41 0.98 0.76
- None 50 1.00 0.74
Preview: a random forest = many trees voting - more accurate, less readable.
Activity: Entropy, Gain and Reading a Tree
CLASS ACTIVITY · PAIRS · 8 MIN
Pairs, 6 min + 2 min check. Calculators allowed. Use the cricket tree of S06 and log2 values from the board.
Entropy of a group with 5 yes, 5 no? With 8 yes, 0 no?
Classify: rain, hot, high humidity, strong wind.
Gini index of 9 yes, 5 no.
From the overfit table (S14), which depth would you choose and why?
Humidity splits the 14 days into high (3 yes, 4 no) and normal (6 yes, 1 no). Entropy of each and Gain(humidity)?
Why does a decision tree not need feature scaling, but k-NN does?
Common Mistakes
WATCH OUT
WRONG IDEA
CORRECT IDEA
✗ High entropy is good.
✓ We want LOW entropy (pure groups) after a split.
✗ Gain = entropy of the biggest branch.
✓ Use the WEIGHTED average of all branches.
✗ The deepest tree is the best tree.
✓ Deep trees overfit; check test accuracy.
✗ Every attribute must appear in the tree.
✓ Useless attributes (temperature) are left out.
✗ A tree finds the best tree overall.
✓ ID3 is greedy: best question at each step only.
Recap Quiz
CHECK YOUR UNDERSTANDING · 3 MIN
- Q1. The entropy of a pure node is:
- (a) 0 (b) 1 (c) 0.5 (d) 0.940
- Q2. ID3 chooses the attribute with the:
- (a) Lowest gain (b) Highest information gain (c) Most values (d) Highest entropy
- Q3. A leaf node holds:
- (a) A question (b) The entropy (c) The predicted class (d) The training data
- Q4. Train 1.00, test 0.74 for a very deep tree shows:
- (a) Underfitting (b) Feature scaling (c) Pruning (d) Overfitting
Homework and Next Lecture
BEFORE THE NEXT CLASS
- Task 1: Entropy of: 6 yes 2 no; 4 yes 4 no; 1 yes 7 no.
- Task 2: 8 loan applicants (job, own house -> A approved / R rejected): govt yes A, govt yes A, private yes A, private yes A, govt no A, private no R, private no R, private no R. Root entropy, Gain(job), Gain(own house), and the full ID3 tree.
- Task 3: Use the cricket tree to classify: (a) sunny, cool, high, strong (b) overcast, hot, high, strong (c) rain, mild, normal, weak.
NEXT LECTURE
- Lecture 26:
- Neural Networks and the Perceptron