Module 2: Language models and transformers · 11 min
Attention and the transformer
Before 2017, language models read text one word at a time and often forgot the start of long sentences. The transformer, introduced in 2017, changed this with a trick called attention.
Attention lets every word look at every other word and decide which ones matter. In the sentence 'The trophy did not fit in the suitcase because it was too big', attention helps the model link 'it' to 'trophy' and not to 'suitcase'.
- Transformers read the whole input in parallel, so they train fast on many chips.
- Stacking many attention layers lets the model build up deeper understanding.
- The model then predicts the next token, adds it to the text, and repeats.
Almost every top language model today is a transformer, and the same idea is also used in image and audio models.
← From words to numbers: tokens and embeddings | Pre-training, fine-tuning and alignment →