Nexus Learn

Log in · Create free account

Module 2: Language models and transformers · 9 min

From words to numbers: tokens and embeddings

Computers work with numbers, so text is first cut into tokens, and each token is turned into a list of numbers called an embedding. Tokens with similar meanings get similar embeddings.

You can imagine each word as a point on a huge map. 'Doctor' and 'nurse' sit close together; 'doctor' and 'banana' sit far apart. The model uses this map to understand relationships.

  • Embeddings capture meaning, not spelling.
  • The same idea powers search by meaning (semantic search), which you will use again in the RAG module.
  • Different languages can share one map, which helps translation.
(free account needed to save progress)

← The main kinds of generative models  |  Attention and the transformer →