Book · Part I · Chapter 2

First there was a token

How computers learned to turn words into numbers: from the telegraph to tokens, and why working with AI costs differently across languages.

Maksim Gurchenkov

Any modern computer is “just” zeros and ones, and any LLM is “just” matrices with nonlinearity between them. On top of all that sit simple math operations and our agreements about how to interpret the whole thing. Since we’ve already covered the zeros, ones, matrices and operations, this chapter and the next are about those agreements.

Neither zeros and ones nor matrices mean anything on their own. A record a machine can read appeared the moment we agreed on how to encode and decode text. And people spent almost two hundred years getting to that agreement on how to encode words as numbers. It didn’t start with AI, or even with computers. It started with the telegraph.

The word TOKEN built from paper fragments and numbers

In this chapter

  1. 2.1 From the telegraph to ASCII Read →
  2. 2.2 From ASCII to the token Read →
  3. 2.3 When the vocabulary became a limit Read →
  4. 2.4 Tokens and languages Read →
Start reading
Get chapter 4 when it’s out