1. Objectives
- Build an XOR network by hand from three step neurons (OR, NAND, AND) and explain why a hidden layer is needed.
- Train scikit-learn's MLPClassifier and explain hidden_layer_sizes, activation, solver, max_iter and random_state.
- Count the weights and biases of a network from its layer sizes (coefs_ and intercepts_).
- Prepare data correctly: train/test split, StandardScaler fitted on training data only.
- Evaluate a classifier with accuracy, a confusion matrix and a learning (loss) curve.
- Choose the number of hidden neurons by experiment and recognise underfitting and overfitting.
2. Word Bank
Word | Simple meaning | Real-world example |
|---|---|---|
Multilayer perceptron (MLP) | A network with one or more hidden layers between input and output | XOR network: 2 inputs → 2 hidden → 1 output |
Hidden layer | Neurons between input and output that make new features | OR and NAND neurons inside the XOR network |
Activation function | The rule that turns z into a neuron's output | step, relu, tanh, logistic (sigmoid) |
Solver | The method that adjusts the weights during training | lbfgs for small data, adam for big data |
Iteration / epoch | One update round of training | n_iter_ = 41 rounds |
Loss | A number that measures how wrong the network is (lower is better) | Loss falls from 0.6 to 0.1 |
Standardise (scale) | Shift and stretch each feature to mean 0, spread 1 | Temperature and humidity on the same scale |
Confusion matrix | Table of correct and wrong predictions per class | Rust days predicted as no rust |
Underfitting | Model too simple - poor on training AND test data | 1 hidden neuron |
Overfitting | Model memorises training data - test score drops | Train 1.0 but test lower |
3. Background
From Practical 2 to today
In Practical 2 the single perceptron learned AND, OR and NAND but failed on XOR: its mistakes never reached zero because no single straight line separates XOR's classes. In Lecture 6 you met XOR as a connective: 'P or Q but not both' = (P ∨ Q) ∧ ¬(P ∧ Q). That formula already tells us the fix: one neuron for OR, one for NAND (= not both), and a third neuron that ANDs them. The first two form a hidden layer.
What the hidden layer does
Each hidden neuron draws its own straight line. The output neuron combines these lines, so the network can draw bent or closed boundaries. More hidden neurons = more flexible boundaries, but also more risk of memorising noise (overfitting).
Real-world link: wheat rust
Yellow (stripe) rust attacks wheat in Punjab when the weather is cool (about 12-22 °C) and very humid. Rust is likely only INSIDE a band of temperature - not too cold, not too hot. A single line cannot describe 'between 12 and 22', so a perceptron struggles; an MLP can learn the band. Today's file wheat_rust.csv has 80 days of temperature, humidity and whether rust appeared (it includes a few noisy, wrongly recorded days, as real field records do).
scikit-learn in one line
MLPClassifier(hidden_layer_sizes=(8,), activation='relu', solver='lbfgs', max_iter=2000, random_state=1).fit(X, y). random_state fixes the random starting weights so everyone in the lab gets the same answer.
Data files for this lab
wheat_rust.csv
day,temp_c,humidity,rust D01,34,74,0 D02,26,93,0 D03,22,85,1 D04,30,45,0 D05,6,51,0 D06,13,92,1 D07,33,30,0 D08,20,88,1 D09,9,86,0 D10,8,63,0 D11,30,51,0 D12,15,49,0 D13,27,48,0 D14,35,61,0 D15,19,65,1 D16,23,69,0 D17,20,100,1 D18,30,86,1 D19,26,74,0 D20,15,100,1 D21,19,45,0 D22,31,41,0 D23,31,73,0 D24,8,33,0 D25,18,32,0 D26,9,66,0 D27,35,63,0 D28,30,95,0 D29,30,74,0 D30,18,66,1 D31,13,65,1 D32,16,47,0 D33,35,30,0 D34,8,43,0 D35,35,79,0 D36,32,44,0 D37,27,56,0 D38,20,30,0 D39,24,88,0 D40,25,40,0 D41,21,48,0 D42,34,92,0 D43,10,66,0 D44,34,90,0 D45,26,75,1 D46,6,82,0 D47,19,36,0 D48,12,68,1 D49,27,66,0 D50,24,91,0 D51,26,55,0 D52,24,72,0 D53,8,34,0 D54,25,57,0 D55,23,52,0 D56,11,40,0 D57,16,87,1 D58,17,56,0 D59,22,99,1 D60,17,71,1 D61,18,72,1 D62,18,75,1 D63,21,78,1 D64,34,40,1 D65,22,61,0 D66,16,47,0 D67,6,58,0 D68,30,36,0 D69,17,98,1 D70,34,45,0 D71,5,77,0 D72,5,51,0 D73,20,92,1 D74,7,77,0 D75,21,39,0 D76,31,90,0 D77,19,97,1 D78,23,94,0 D79,34,70,0 D80,13,40,0 |
4. Time Plan (180 minutes)
Minutes | Activity | Part |
|---|---|---|
0-15 | XOR by hand with three step neurons | Part A1 |
15-60 | MLPClassifier on XOR; wheat rust: perceptron vs MLP; confusion matrix; loss curve | Part A2-A5 |
60-75 | Predict the output | Part B |
75-95 | Bug hunt | Part C |
95-165 | Graded tasks 1-5 | Part D |
165-180 | Post-lab questions, lab book entry, submission | Part E |
Part A - Guided Warm-up (type, run, observe)
A1. XOR by hand: a hidden layer of two neurons
Hidden neuron h1 is OR (w = [1, 1], b = -0.5). Hidden neuron h2 is NAND (w = [-1, -1], b = 1.5). The output neuron is AND of h1 and h2 (w = [1, 1], b = -1.5). This is exactly XOR = (P ∨ Q) ∧ ¬(P ∧ Q) from Lecture 6.
import numpy as np step = lambda z: (z >= 0).astype(int) X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) h1 = step(X @ np.array([1, 1]) - 0.5) # OR h2 = step(X @ np.array([-1, -1]) + 1.5) # NAND H = np.column_stack([h1, h2]) # hidden layer output out = step(H @ np.array([1, 1]) - 1.5) # AND of h1 and h2 print("x1 x2 | h1(OR) h2(NAND) | XOR") for (a, b), p, q, o in zip(X, h1, h2, out): print(f" {a} {b} | {p} {q} | {o}") |
Output x1 x2 | h1(OR) h2(NAND) | XOR 0 0 | 0 1 | 0 0 1 | 1 1 | 1 1 0 | 1 1 | 1 1 1 | 1 0 | 0 |
A2. Let the network learn XOR (MLPClassifier)
Now we do not choose the weights. hidden_layer_sizes=(4,) means one hidden layer of 4 neurons. lbfgs is a good solver for very small data sets.
import numpy as np from sklearn.neural_network import MLPClassifier X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) y = np.array([0, 1, 1, 0]) mlp = MLPClassifier(hidden_layer_sizes=(4,), activation="tanh", solver="lbfgs", max_iter=1000, random_state=1) mlp.fit(X, y) print("predictions:", mlp.predict(X), " true:", y) print("training rounds (n_iter_):", mlp.n_iter_) print("weight matrix shapes:", [w.shape for w in mlp.coefs_]) print("bias vector shapes :", [b.shape for b in mlp.intercepts_]) |
Output predictions: [0 1 1 0] true: [0 1 1 0] training rounds (n_iter_): 29 weight matrix shapes: [(2, 4), (4, 1)] bias vector shapes : [(4,), (1,)] |
A3. Wheat rust: perceptron vs multilayer network
Split the data into 75% training and 25% testing (stratify keeps the same share of rust days in both). Fit the scaler on the TRAINING data only, then use it on both parts.
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import Perceptron from sklearn.neural_network import MLPClassifier df = pd.read_csv("wheat_rust.csv") X = df[["temp_c", "humidity"]].to_numpy(dtype=float) y = df["rust"].to_numpy() print("days:", len(df), "| rust days:", y.sum()) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) scaler = StandardScaler().fit(X_train) X_train_s, X_test_s = scaler.transform(X_train), scaler.transform(X_test) per = Perceptron(random_state=0).fit(X_train_s, y_train) mlp = MLPClassifier(hidden_layer_sizes=(8,), solver="lbfgs", max_iter=2000, random_state=1).fit(X_train_s, y_train) print(f"Perceptron train {per.score(X_train_s, y_train):.3f} test {per.score(X_test_s, y_test):.3f}") print(f"MLP (8,) train {mlp.score(X_train_s, y_train):.3f} test {mlp.score(X_test_s, y_test):.3f}") |
Output days: 80 | rust days: 21 Perceptron train 0.733 test 0.750 MLP (8,) train 0.917 test 0.950 |
A4. Confusion matrix: which mistakes does it make?
Accuracy alone hides WHICH days are wrong. For a farmer, missing a rust day (no spray, crop damaged) is worse than a false alarm (one extra spray).
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.neural_network import MLPClassifier from sklearn.metrics import confusion_matrix df = pd.read_csv("wheat_rust.csv") X = df[["temp_c", "humidity"]].to_numpy(dtype=float); y = df["rust"].to_numpy() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) scaler = StandardScaler().fit(X_train) mlp = MLPClassifier(hidden_layer_sizes=(8,), solver="lbfgs", max_iter=2000, random_state=1) mlp.fit(scaler.transform(X_train), y_train) pred = mlp.predict(scaler.transform(X_test)) cm = confusion_matrix(y_test, pred) print("rows = true (0 no rust, 1 rust), columns = predicted") print(cm) print("missed rust days (false negatives):", cm[1, 0]) print("false alarms (false positives) :", cm[0, 1]) |
Output rows = true (0 no rust, 1 rust), columns = predicted [[14 1] [ 0 5]] missed rust days (false negatives): 0 false alarms (false positives) : 1 |
A5. Watching the network learn: the loss curve
The adam solver records the loss after every round in loss_curve_. A falling curve means the network is learning.
import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.neural_network import MLPClassifier df = pd.read_csv("wheat_rust.csv") X = df[["temp_c", "humidity"]].to_numpy(dtype=float); y = df["rust"].to_numpy() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) scaler = StandardScaler().fit(X_train) mlp = MLPClassifier(hidden_layer_sizes=(8,), solver="adam", learning_rate_init=0.01, max_iter=300, random_state=1) mlp.fit(scaler.transform(X_train), y_train) plt.figure(figsize=(6.5, 3.4)) plt.plot(mlp.loss_curve_) plt.xlabel("Training round"); plt.ylabel("Loss"); plt.title("MLP learning curve (adam)") plt.grid(alpha=0.3); plt.tight_layout(); plt.savefig("loss_curve.png", dpi=100) print("first loss:", round(mlp.loss_curve_[0], 3), "| last loss:", round(mlp.loss_curve_[-1], 3)) print("test accuracy:", round(mlp.score(scaler.transform(X_test), y_test), 3)) print("Saved loss_curve.png") |
Output first loss: 0.858 | last loss: 0.245 test accuracy: 0.95 Saved loss_curve.png |
Part B - Predict the Output
Write your prediction BEFORE running the code. Then run it and correct yourself in a different colour.
B1.
import numpy as np from sklearn.neural_network import MLPClassifier X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]); y = np.array([0, 1, 1, 0]) m = MLPClassifier(hidden_layer_sizes=(5, 3), solver="lbfgs", max_iter=500, random_state=0).fit(X, y) print([w.shape for w in m.coefs_]) |
My prediction |
B2.
layers = [2, 5, 3, 1] weights = sum(a * b for a, b in zip(layers, layers[1:])) biases = sum(layers[1:]) print(weights, biases, weights + biases) |
My prediction |
B3.
import numpy as np z = np.array([-2.0, 0.0, 3.0]) print(np.maximum(0, z)) # relu print(np.tanh(0.0), 1 / (1 + np.exp(-0.0))) # tanh(0), logistic(0) |
My prediction |
B4.
import numpy as np from sklearn.preprocessing import StandardScaler x = np.array([[2.0], [4.0], [6.0]]) print(StandardScaler().fit_transform(x).round(4).ravel()) |
My prediction |
B5.
from sklearn.metrics import confusion_matrix y_true = [1, 0, 1, 1, 0] y_pred = [1, 1, 1, 0, 0] print(confusion_matrix(y_true, y_pred)) |
My prediction |
Part C - Bug Hunt
Each program has ONE bug. Some crash with an error message; others run but give a wrong answer (silent logic error - the dangerous kind). Find the bug, explain it and write the fix.
C1.
import numpy as np from sklearn.neural_network import MLPClassifier temps = np.array([15, 30, 18, 33]) # one feature rust = np.array([1, 0, 1, 0]) m = MLPClassifier(solver="lbfgs", random_state=1).fit(temps, rust) |
Bug, why it happens, and the fixed line(s) |
C2.
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier from sklearn.metrics import accuracy_score df = pd.read_csv("wheat_rust.csv") X = df[["temp_c", "humidity"]].to_numpy(dtype=float); y = df["rust"].to_numpy() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) m = MLPClassifier(solver="lbfgs", max_iter=2000, random_state=1).fit(X_train, y_train) print(accuracy_score(X_test, m.predict(X_test))) |
Bug, why it happens, and the fixed line(s) |
C3.
import numpy as np from sklearn.neural_network import MLPClassifier X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]); y = np.array([0, 1, 1, 0]) m = MLPClassifier(hidden_layer_sizes=(4,), activation="identity", solver="lbfgs", max_iter=1000, random_state=1).fit(X, y) print("XOR predictions:", m.predict(X)) |
It runs and prints XOR predictions: [0 0 0 0] |
Bug, why it happens, and the fixed line(s) |
C4.
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.neural_network import MLPClassifier df = pd.read_csv("wheat_rust.csv") X = df[["temp_c", "humidity"]].to_numpy(dtype=float); y = df["rust"].to_numpy() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) scaler = StandardScaler().fit(X_train) m = MLPClassifier(hidden_layer_sizes=(8,), solver="lbfgs", max_iter=2000, random_state=1) m.fit(scaler.transform(X_train), y_train) print("test accuracy:", m.score(X_test, y_test)) # evaluate |
It runs and prints test accuracy: 0.65 |
Bug, why it happens, and the fixed line(s) |
C5.
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.neural_network import MLPClassifier df = pd.read_csv("wheat_rust.csv") X = df[["temp_c", "humidity"]].to_numpy(dtype=float); y = df["rust"].to_numpy() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y) scaler = StandardScaler().fit(X_train) m = MLPClassifier(hidden_layer_sizes=(16,), solver="lbfgs", max_iter=2000, random_state=1) m.fit(scaler.transform(X_train), y_train) print("test accuracy:", m.score(scaler.transform(X_train), y_train)) # report to the farmer |
It runs and prints test accuracy: 1.0 |
Bug, why it happens, and the fixed line(s) |
Part D - Graded Tasks (20 marks)
Task | Level | Title | Marks |
|---|---|---|---|
T1 | Easy | XNOR network by hand | 3 |
T2 | Medium | MLP learns XOR and XNOR; count its parameters | 4 |
T3 | Medium | Wheat rust: perceptron vs MLP with a confusion matrix | 4 |
T4 | Medium | How many hidden neurons? Underfitting and overfitting | 4 |
T5 | Challenge | Rust warning system for three days | 5 |
Total | 20 |
Task 1 (Easy, 3 marks): XNOR network by hand
XNOR gives 1 when both inputs are the SAME. Build it by hand (no training) with three step neurons: hidden h1 = AND (w = [1, 1], b = -1.5), hidden h2 = NOR (choose your own weights), output = OR of h1 and h2. Print the full table with h1, h2 and the output.
Sample run (your output must match):
Output x1 x2 | h1(AND) h2(NOR) | XNOR 0 0 | 0 1 | 1 0 1 | 0 0 | 0 1 0 | 0 0 | 0 1 1 | 1 0 | 1 |
Your code (or attach the file) |
Task 2 (Medium, 4 marks): MLP learns XOR and XNOR; count its parameters
Train an MLPClassifier (hidden_layer_sizes=(4,), activation='tanh', solver='lbfgs', max_iter=1000, random_state=1) on XOR and on XNOR. For each print the predictions and n_iter_. Then print the total number of weights and biases of the network using coefs_ and intercepts_, and check it against the formula 2·4 + 4·1 weights + 4 + 1 biases.
Sample run (your output must match):
Output XOR predictions: [0 1 1 0] rounds: 29 XNOR predictions: [1 0 0 1] rounds: 62 weights: 12 | biases: 5 | total: 17 | formula: 17 |
Your code (or attach the file) |
Task 3 (Medium, 4 marks): Wheat rust: perceptron vs MLP with a confusion matrix
Using wheat_rust.csv with the same split as Part A3 (test_size=0.25, random_state=42, stratify=y) and a StandardScaler fitted on the training data: train a Perceptron (random_state=0) and an MLPClassifier (hidden_layer_sizes=(8,), solver='lbfgs', max_iter=2000, random_state=1). Print the test accuracy of both and the confusion matrix of each. In one line, say which model a farmer should trust and why.
Sample run (your output must match):
Output Perceptron: test accuracy 0.750 [[15 0] [ 5 0]] MLP (8,): test accuracy 0.950 [[14 1] [ 0 5]] Trust the MLP: it misses fewer rust days because it can learn the 12-22 °C band. |
Your code (or attach the file) |
Task 4 (Medium, 4 marks): How many hidden neurons? Underfitting and overfitting
With the same split and scaler, train MLPClassifier(hidden_layer_sizes=(h,), solver='lbfgs', max_iter=2000, random_state=1) for h = 1, 2, 4, 8, 16. Print the training and test accuracy for each, plot both against h (save task4.png) and write which h you would choose and why.
Sample run (your output must match):
Output h=1 train 0.733 test 0.750 h=2 train 0.733 test 0.750 h=4 train 0.900 test 0.900 h=8 train 0.917 test 0.950 h=16 train 1.000 test 0.950 Saved task4.png |
Your code (or attach the file) |
Task 5 (Challenge, 5 marks): Rust warning system for three days
Build a Pipeline of StandardScaler and MLPClassifier(hidden_layer_sizes=(8,), solver='lbfgs', max_iter=2000, random_state=1). Train it on ALL 80 days. For the forecast days (temp 16 °C, humidity 88%), (30 °C, 85%) and (14 °C, 50%), print the probability of rust (predict_proba) and the advice 'SPRAY' if the probability is 0.5 or more, otherwise 'no spray'. In two lines, explain why day 2 and day 3 get 'no spray' even though one is very humid and the other is cool.
Sample run (your output must match):
Output 16 °C, 88% humidity -> rust probability 1.00 -> SPRAY 30 °C, 85% humidity -> rust probability 0.03 -> no spray 14 °C, 50% humidity -> rust probability 0.00 -> no spray |
Your code (or attach the file) |
Part E - Post-Lab Questions
E1. Why can a network with a hidden layer learn XOR when a single perceptron cannot?
Your answer |
E2. A network has layer sizes [2, 8, 1]. How many weights and biases does it have?
Your answer |
E3. Why must the scaler be fitted on the training data only, and applied to the test data too?
Your answer |
E4. In Task 4, what happened with 1 hidden neuron and with 16? Name both problems.
Your answer |
E5. For the rust warning system, which is worse: a false negative or a false positive? Why?
Your answer |
Marking Rubric (applied to Part D)
Criterion | Full | Partial | None |
|---|---|---|---|
Code runs without errors | Runs fully | Small fixable error | Does not run |
Correct data preparation (split, scaler on training data) | Correct | Minor slip | Leaks or missing |
Correct model settings and results | All match | Most match | Few or none |
Plots and printed results labelled | All present | Some missing | None |
Explanations (hidden layer, under/overfitting, advice) | Clear and correct | Partly correct | Missing |
Submission
- Make a folder RollNo_P03 with task1.py ... task5.py (or one notebook P03.ipynb), wheat_rust.csv and task4.png.
- Zip the folder and upload it to the LMS / Google Classroom by the end of the lab day.
- Copy Task 3 and Task 4 code and output into your Lab Book (Practical 3 pages) for signature next week.
- Use random_state exactly as given so your numbers match the sample runs.