Nexus Learn

Lecture 5: Benchmarks and Metrics

Lecture Slides CS-704 Advanced Computer Architecture 6 min read Updated 8 Nov 2026

↓ Download slides (PowerPoint)

Recap of Lecture 4 and Homework Check

  • Performance = 1 / execution time.
  • CPU time = IC x CPI / clock rate.
  • Average CPI = sum of (fraction x CPI) over instruction classes.
  • Fewer instructions is not always faster - count the cycles.

Homework check

  • Classes A, B, C: CPI 1, 2, 3.
  • Seq 1: 4 A, 2 B, 1 C
  • Seq 2: 1 A, 1 B, 3 C
  • IC, cycles, CPI? Faster?
  • Practical 2: add-only 28 instr; mul version 29 (2 mul at 4 cycles).

Today's Plan (50 minutes)

0-5

Recap and homework check Lecture 4

5-8

Plan and learning outcomes

8-17

Module 1: What is a benchmark? kinds and suites (SPEC, TPC, EEMBC)

17-27

Module 2: Misleading metrics MIPS, MFLOPS, peak performance

27-36

Module 3: Summarizing results arithmetic vs geometric mean

36-44

Class activity four short numericals

44-50

Mistakes, quiz and homework

Learning Outcomes (SLOs)

By the end of this lecture you will be able to:

SLO 5.1

Classify benchmarks (real programs, kernels, toy, synthetic, suites) and explain why suites of real programs are preferred.

→ CLO2 · C2 Understand

SLO 5.2

Calculate MIPS and show with an example why MIPS, MFLOPS and peak performance can mislead.

→ CLO2 · C3 Apply

SLO 5.3

Summarize results with SPECratios and the geometric mean, and choose the right mean for a situation.

→ CLO2 · C3 Apply

CLO2: Measure and report computing performance using benchmarks, metrics, the CPU performance equation and Amdahl's law.

Module 1: What Is a Benchmark?

A program chosen to measure and compare computers

Why Do We Need Benchmarks?

Phone reviews

Fuel average

  • Review sites show AnTuTu or Geekbench scores to compare phones.
  • A benchmark = a standard program run on every machine.
  • Company claims 20 km/l (test track).
  • In Lahore traffic you get 13 km/l.
  • The test must look like real use.

Best test: your own workload (the programs you really run). Next best: standard suites of real programs.

Kinds of Benchmarks

Why synthetic fails: compiler writers learned to optimize the benchmark itself - the score rose, real programs did not.

Benchmark Suites in Use Today

SPEC CPU (desktop and server)

TPC (databases)

Real programs, run since 1989. SPEC CPU 2026 (May 2026): 52 benchmarks; replaces CPU 2017.

  • TPC-C: online orders (like Daraz).
  • TPC-H: business reports.

EEMBC (embedded)

MLPerf (AI)

CoreMark for processors in cars, routers and washing machines.

Time to train or run AI models on a system.

Module 2: Misleading Metrics

Numbers that look like performance but are not

MIPS: Million Instructions Per Second

  • MIPS = Instruction count / (Execution time x 10^6)
  • = Clock rate / (CPI x 10^6)
  • Example: 2 GHz, CPI 1.25 -> 2000 / 1.25 = 1600 MIPS

Ignores the ISA

Changes per program

Can go the wrong way

One CISC instruction can do the work of five RISC ones.

No single MIPS value for a computer.

Higher MIPS, yet slower (next slide).

Worked Example: Higher MIPS, Slower Program

  • Compiler 1: time = 10e9 / 4e9 = 2.5 s MIPS = 7e9 / (2.5 x 10^6) = 2800
  • Compiler 2: time = 15e9 / 4e9 = 3.75 s MIPS = 12e9 / (3.75 x 10^6) = 3200

Result: Compiler 2 has higher MIPS but is 1.5 times slower (3.75 / 2.5). Judge by time, never by MIPS.

MFLOPS and Peak Performance

MFLOPS

Peak performance

  • Millions of floating-point operations per second.
  • Ignores integer work; counts a divide the same as an add.
  • The most the hardware could do: 3 GHz x 4 per cycle = 12,000 MIPS.
  • Real program, CPI 1.2: 2,500 MIPS.

21%

Like a car: the speedometer says 220 km/h; your real average to Lahore is about 90 km/h.

of peak achieved

2,500 / 12,000

Module 3: Summarizing Results

Many programs, one fair number

Arithmetic Mean of Times

  • Arithmetic mean AM = (1/n) x sum(Time_i)
  • Weighted mean WAM = sum(Weight_i x Time_i), weights add to 1

Good when

Weak when

You know the real workload: total time is the truth.

Weights are unknown, or one long program dominates the mean.

Example: a bank runs transactions 70% and reports 30% of the day: WAM = 0.7 x T1 + 0.3 x T2.

The Trap: Averaging Normalized Times

Lesson: the arithmetic mean of ratios changes with the reference machine. The geometric mean does not.

Total time for 'run each once': A = 1001 s, B = 110 s - B is 9.1 times faster for that workload.

SPECratio and the Geometric Mean

  • ratio = Ref / Time
  • GM = (r1 x..x rn)^(1/n)
  • A: 600^(1/3) = 8.43
  • B: 640^(1/3) = 8.62

Result: B is 8.62 / 8.43 = 1.02 times faster by SPEC's measure. (Times are made up for practice.)

SPECratio: how many times faster than the reference machine. Bigger is better.

Why the Geometric Mean? And Its Limit

Strength

Limit

  • GM(A) / GM(B) = GM(time ratios).
  • The reference machine cancels: same answer for any reference.
  • GM does not give total time.
  • Total: A 430 s, B 375 s - B is 1.15x faster (not 1.02x).

Class Activity: Four Quick Numericals

7 min · pairs

Solve in pairs with a calculator. Show the formula you use.

A 2 GHz processor runs a program with CPI 1.25. Find its MIPS.

A program of 2 x 10^9 instructions runs in 1.6 s. Find its MIPS.

SPECratios on three programs are 4, 9 and 6. Find the geometric mean.

X has ratios 2 and 8; Y has ratios 4 and 4. Compare them with the arithmetic and the geometric mean.

Common Mistakes

Wrong idea

Correct idea

✗ A higher MIPS rating means a faster computer.

✓ MIPS ignores the ISA and the work per instruction; compare time.

✗ Peak performance tells real speed.

✓ Peak is a never-exceed limit; real programs reach a fraction.

✗ Synthetic benchmarks represent real work.

✓ They are easy to cheat; real programs in a suite are trusted.

✗ Average the normalized times with an arithmetic mean.

✓ That depends on the reference; use the geometric mean of ratios.

Recap Quiz

Answer on paper in 3 minutes, then we check together.

Q1

Name the four kinds of benchmarks, from most to least trusted.

Q2

Write the formula for MIPS in terms of clock rate and CPI.

Q3

Give one reason why MIPS can mislead.

Q4

Define SPECratio. Is a bigger SPECratio better or worse?

Q5

SPECratios 2, 8 and 4: find the geometric mean.

Homework and Next Lecture

  • Read Hennessy and Patterson, Chapter 1, section 1.8 (benchmarks and summarizing results).
  • MIPS trap: 3 GHz machine; classes A, B, C have CPI 1, 2, 4. Compiler X: 6 billion A, 2 billion B, 1 billion C. Compiler Y: 4 billion A, 1 billion B, 2 billion C. Find time and MIPS of each. Which is faster? Which has the higher MIPS?
  • SPEC: times (s) on the reference: 900, 1200, 600. Machine P: 90, 100, 75. Machine Q: 60, 150, 60. Find SPECratios and GM of each; which is faster by SPEC?

NEXT LECTURE

Lecture 6: Quantitative Principles of Design

Parallelism, the principle of locality, focusing on the common case, and Amdahl's law with worked problems.

← All CS-704 material