Recap of Lecture 4 and Homework Check
- Performance = 1 / execution time.
- CPU time = IC x CPI / clock rate.
- Average CPI = sum of (fraction x CPI) over instruction classes.
- Fewer instructions is not always faster - count the cycles.
Homework check
- Classes A, B, C: CPI 1, 2, 3.
- Seq 1: 4 A, 2 B, 1 C
- Seq 2: 1 A, 1 B, 3 C
- IC, cycles, CPI? Faster?
- Practical 2: add-only 28 instr; mul version 29 (2 mul at 4 cycles).
Today's Plan (50 minutes)
0-5
Recap and homework check Lecture 4
5-8
Plan and learning outcomes
8-17
Module 1: What is a benchmark? kinds and suites (SPEC, TPC, EEMBC)
17-27
Module 2: Misleading metrics MIPS, MFLOPS, peak performance
27-36
Module 3: Summarizing results arithmetic vs geometric mean
36-44
Class activity four short numericals
44-50
Mistakes, quiz and homework
Learning Outcomes (SLOs)
By the end of this lecture you will be able to:
SLO 5.1
Classify benchmarks (real programs, kernels, toy, synthetic, suites) and explain why suites of real programs are preferred.
→ CLO2 · C2 Understand
SLO 5.2
Calculate MIPS and show with an example why MIPS, MFLOPS and peak performance can mislead.
→ CLO2 · C3 Apply
SLO 5.3
Summarize results with SPECratios and the geometric mean, and choose the right mean for a situation.
→ CLO2 · C3 Apply
CLO2: Measure and report computing performance using benchmarks, metrics, the CPU performance equation and Amdahl's law.
Module 1: What Is a Benchmark?
A program chosen to measure and compare computers
Why Do We Need Benchmarks?
Phone reviews
Fuel average
- Review sites show AnTuTu or Geekbench scores to compare phones.
- A benchmark = a standard program run on every machine.
- Company claims 20 km/l (test track).
- In Lahore traffic you get 13 km/l.
- The test must look like real use.
Best test: your own workload (the programs you really run). Next best: standard suites of real programs.
Kinds of Benchmarks
Why synthetic fails: compiler writers learned to optimize the benchmark itself - the score rose, real programs did not.
Benchmark Suites in Use Today
SPEC CPU (desktop and server)
TPC (databases)
Real programs, run since 1989. SPEC CPU 2026 (May 2026): 52 benchmarks; replaces CPU 2017.
- TPC-C: online orders (like Daraz).
- TPC-H: business reports.
EEMBC (embedded)
MLPerf (AI)
CoreMark for processors in cars, routers and washing machines.
Time to train or run AI models on a system.
Module 2: Misleading Metrics
Numbers that look like performance but are not
MIPS: Million Instructions Per Second
- MIPS = Instruction count / (Execution time x 10^6)
- = Clock rate / (CPI x 10^6)
- Example: 2 GHz, CPI 1.25 -> 2000 / 1.25 = 1600 MIPS
Ignores the ISA
Changes per program
Can go the wrong way
One CISC instruction can do the work of five RISC ones.
No single MIPS value for a computer.
Higher MIPS, yet slower (next slide).
Worked Example: Higher MIPS, Slower Program
- Compiler 1: time = 10e9 / 4e9 = 2.5 s MIPS = 7e9 / (2.5 x 10^6) = 2800
- Compiler 2: time = 15e9 / 4e9 = 3.75 s MIPS = 12e9 / (3.75 x 10^6) = 3200
Result: Compiler 2 has higher MIPS but is 1.5 times slower (3.75 / 2.5). Judge by time, never by MIPS.
MFLOPS and Peak Performance
MFLOPS
Peak performance
- Millions of floating-point operations per second.
- Ignores integer work; counts a divide the same as an add.
- The most the hardware could do: 3 GHz x 4 per cycle = 12,000 MIPS.
- Real program, CPI 1.2: 2,500 MIPS.
21%
Like a car: the speedometer says 220 km/h; your real average to Lahore is about 90 km/h.
of peak achieved
2,500 / 12,000
Module 3: Summarizing Results
Many programs, one fair number
Arithmetic Mean of Times
- Arithmetic mean AM = (1/n) x sum(Time_i)
- Weighted mean WAM = sum(Weight_i x Time_i), weights add to 1
Good when
Weak when
You know the real workload: total time is the truth.
Weights are unknown, or one long program dominates the mean.
Example: a bank runs transactions 70% and reports 30% of the day: WAM = 0.7 x T1 + 0.3 x T2.
The Trap: Averaging Normalized Times
Lesson: the arithmetic mean of ratios changes with the reference machine. The geometric mean does not.
Total time for 'run each once': A = 1001 s, B = 110 s - B is 9.1 times faster for that workload.
SPECratio and the Geometric Mean
- ratio = Ref / Time
- GM = (r1 x..x rn)^(1/n)
- A: 600^(1/3) = 8.43
- B: 640^(1/3) = 8.62
Result: B is 8.62 / 8.43 = 1.02 times faster by SPEC's measure. (Times are made up for practice.)
SPECratio: how many times faster than the reference machine. Bigger is better.
Why the Geometric Mean? And Its Limit
Strength
Limit
- GM(A) / GM(B) = GM(time ratios).
- The reference machine cancels: same answer for any reference.
- GM does not give total time.
- Total: A 430 s, B 375 s - B is 1.15x faster (not 1.02x).
Class Activity: Four Quick Numericals
7 min · pairs
Solve in pairs with a calculator. Show the formula you use.
A 2 GHz processor runs a program with CPI 1.25. Find its MIPS.
A program of 2 x 10^9 instructions runs in 1.6 s. Find its MIPS.
SPECratios on three programs are 4, 9 and 6. Find the geometric mean.
X has ratios 2 and 8; Y has ratios 4 and 4. Compare them with the arithmetic and the geometric mean.
Common Mistakes
Wrong idea
Correct idea
✗ A higher MIPS rating means a faster computer.
✓ MIPS ignores the ISA and the work per instruction; compare time.
✗ Peak performance tells real speed.
✓ Peak is a never-exceed limit; real programs reach a fraction.
✗ Synthetic benchmarks represent real work.
✓ They are easy to cheat; real programs in a suite are trusted.
✗ Average the normalized times with an arithmetic mean.
✓ That depends on the reference; use the geometric mean of ratios.
Recap Quiz
Answer on paper in 3 minutes, then we check together.
Q1
Name the four kinds of benchmarks, from most to least trusted.
Q2
Write the formula for MIPS in terms of clock rate and CPI.
Q3
Give one reason why MIPS can mislead.
Q4
Define SPECratio. Is a bigger SPECratio better or worse?
Q5
SPECratios 2, 8 and 4: find the geometric mean.
Homework and Next Lecture
- Read Hennessy and Patterson, Chapter 1, section 1.8 (benchmarks and summarizing results).
- MIPS trap: 3 GHz machine; classes A, B, C have CPI 1, 2, 4. Compiler X: 6 billion A, 2 billion B, 1 billion C. Compiler Y: 4 billion A, 1 billion B, 2 billion C. Find time and MIPS of each. Which is faster? Which has the higher MIPS?
- SPEC: times (s) on the reference: 900, 1200, 600. Machine P: 90, 100, 75. Machine Q: 60, 150, 60. Find SPECratios and GM of each; which is faster by SPEC?
NEXT LECTURE
Lecture 6: Quantitative Principles of Design
Parallelism, the principle of locality, focusing on the common case, and Amdahl's law with worked problems.