Nexus Learn

Log in · Create free account

Module 2: Language models and transformers · 11 min

Attention and the transformer

Before 2017, language models read text one word at a time and often forgot the start of long sentences. The transformer, introduced in 2017, changed this with a trick called attention.

Attention lets every word look at every other word and decide which ones matter. In the sentence 'The trophy did not fit in the suitcase because it was too big', attention helps the model link 'it' to 'trophy' and not to 'suitcase'.

  • Transformers read the whole input in parallel, so they train fast on many chips.
  • Stacking many attention layers lets the model build up deeper understanding.
  • The model then predicts the next token, adds it to the text, and repeats.

Almost every top language model today is a transformer, and the same idea is also used in image and audio models.

(free account needed to save progress)

← From words to numbers: tokens and embeddings  |  Pre-training, fine-tuning and alignment →