The task is familiar from Shannon’s 1951 experiment1: a person saw the beginning of a text and guessed the next letter. The model does the same thing with tokens. Only instead of human intuition, it has a vector gathered from everything written so far.
At the very end, this vector is multiplied by one more matrix, and the result is one score for each token in the vocabulary: about two hundred thousand tokens, about two hundred thousand numbers. These scores are called logits. The higher the logit, the better the token fits as the next one. Logits aren’t probabilities yet: they can be negative and can add up to anything. The softmax function turns them into shares that add up to 100%. What it does is simple: it raises the number e, about 2.72, to the power of each score and divides by the total.
Take the question “Which camera should I buy for a blog?” The answer doesn’t have to start with “Sony” or “Canon.” It could start with “For,” with “It” or with a question back. Say these three openings have logits of 2, 1 and 0, and for simplicity we drop the rest of the vocabulary:
logit e to the power share
For 2 7.39 67%
It 1 2.72 24%
What 0 1.00 9%
total 11.11 100%
You can check it on a calculator. A difference of one unit of logit turns into a nearly threefold difference in probability. “For a beginner…”, “It depends…” and “What is your budget?” lead to three different answers, and each opening makes sense.
Which token should we take? The most obvious choice: always the most likely one. People tried. In 2019 Ari Holtzman and colleagues2 showed where that leads: a model that always takes the top of the distribution gets stuck in a loop after a couple of paragraphs and repeats the same phrase over and over. So the next token is chosen with a bit of randomness: a likely token has better odds, but it doesn’t always win. This is called sampling. Holtzman proposed drawing lots only among sufficiently likely candidates and cutting off the long tail of the rest, and this method, nucleus sampling, is still in use. Randomness here isn’t a bug but a cure for looping. Because of it, the same question asked twice can get different answers starting from the very first word.
The amount of randomness is controlled by a setting called temperature. Before softmax, all the logits are divided by it:
temperature For It What
0.5 87% 12% 2%
1 67% 24% 9%
2 51% 31% 19%
Low temperature strengthens the leader, high temperature gives the others a chance. It’s a dial for variety, not for truthfulness: at low temperature the model will repeat the same mistake word for word.
The chosen token is appended to the sequence, and everything repeats: the model computes the next distribution, now taking the appended token into account. Then another one. An answer to the camera question is hundreds of such steps in a row, until a special end-of-answer token comes up.
Text
↓
Tokens
↓
Vectors and model layers ◀── ①
↓
Logits
↓
Probabilities
↓
Token choice ──▶ ① append, repeat
↓
Answer
In Transformer Explainer3 you can watch this whole loop on a small working model right in your browser: tokens, attention, probabilities, choice. And the real percentages of big models are shown by the logprobs mode4 in their APIs.
Using an already trained model is called inference. The weights don’t change during it: only the input text changes.
It all seems like “just autocomplete.” Yes and no. The mechanics really are simple. But to predict the next token well across textbooks, code, conversations and camera reviews all at once, the model had to learn grammar, facts, styles and the connections between things. Predicting the next token is the mechanics. The behavior that grows out of it is far more complex. Chapter 1 had the same story: simple operations plus scale produce surprisingly complex properties.
Takeaway 1. The probability of the next token is not the probability that a sentence is true, and not the probability that a company will be recommended. “I recommend X” and “I would avoid X” both contain X. Only a whole answer tells you whether something is recommended, and because of sampling it’s a little different every time. So you can judge what a model says about a company from many answers, not from one.
1951 Shannon: language can be grasped by guessing the next piece
Logits and softmax: the vector turns into probabilities for every token in the vocabulary
2019 Holtzman: you can't always take the most likely token: the text loops
Generation: pick a token, append it, compute the next one
The circle is closed. Text became IDs, IDs became vectors, vectors got to know their neighbors, and the next word fell out of the last vector. Along the way there was nothing but simple operations on numbers, and yet on the screen there’s coherent advice about a camera. To me, that’s astonishing.
But why are the numbers in these matrices exactly what they are? Why does a question about a camera produce a meaningful answer and not a random string of characters? Where does the model know all this from?
Sources
-
Claude Shannon, “Prediction and Entropy of Printed English,” 1951. Predicting the next letter as a way to measure the patterns of a language. — https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf ↩
-
Holtzman et al., “The Curious Case of Neural Text Degeneration,” 2019, ICLR 2020. Why the next token is chosen with a bit of randomness, and nucleus sampling. — https://arxiv.org/abs/1904.09751 ↩
-
Transformer Explainer, Georgia Tech. An interactive walk from tokens to the probabilities of what comes next. — https://poloclub.github.io/transformer-explainer/ ↩
-
OpenAI Cookbook, “Using logprobs.” How to see next-token probabilities in the API. — https://cookbook.openai.com/examples/using_logprobs ↩