Module 2: Language models and transformers · 9 min
From words to numbers: tokens and embeddings
Computers work with numbers, so text is first cut into tokens, and each token is turned into a list of numbers called an embedding. Tokens with similar meanings get similar embeddings.
You can imagine each word as a point on a huge map. 'Doctor' and 'nurse' sit close together; 'doctor' and 'banana' sit far apart. The model uses this map to understand relationships.
- Embeddings capture meaning, not spelling.
- The same idea powers search by meaning (semantic search), which you will use again in the RAG module.
- Different languages can share one map, which helps translation.
← The main kinds of generative models | Attention and the transformer →