The vector from the embedding table is a dictionary vector. It’s the same for “mouse” in any text. “The cat chased the mouse” and “The laptop needs a mouse” end with the same word, but the mice are different. A model that worked only with dictionary vectors would know words but wouldn’t understand sentences. The vector needs to find out who’s standing next to it.
Neural networks took a long time to learn to read a sentence as a whole. The first networks for text read it word by word, the way a person runs a finger along a line, and carried a single memory vector with them, adding each new word to it. By the end of a long sentence, almost nothing of its beginning was left in that memory. And such a network was hard to train: the error signal that backpropagation rolls backward faded on the way from the end of the sentence to the start. In 1997 Sepp Hochreiter and Jürgen Schmidhuber1 gave the network a better memory. Their LSTM is a separate cell that decides for itself what to remember for a long time and what to forget. With it, networks got decent at speech recognition and translation, and until 2017 almost all neural network work with text rested on LSTMs. But it still read in order: until the tenth word is processed, you can’t start on the eleventh.
Translators had an extra problem on top of that. A 2014 neural machine translator worked like this: one network reads the sentence and squeezes it into a single vector, a second one unfolds the translation from that vector. On short sentences it worked. On long ones quality collapsed2: the whole sentence had to be pushed through one narrow bottleneck. That same year Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio3 proposed a simple thing: let the translator, as it picks each word of the translation, look back at the original and decide for itself which words to look at right now. The mechanism was called attention. Two years later Google Translate switched to such a system4, and translation quality jumped so much that newspapers wrote about it. But attention was still an add-on to reading in order, a crutch for long sentences.
Then in 2017 came what is probably the most famous paper in the history of AI: “Attention Is All You Need”5. Its eight authors at Google had a problem of their own. Reading in order didn’t just forget things, it also couldn’t be split into parts. GPUs, which all of deep learning had relied on since AlexNet, can perform thousands of operations at once, but here they had to wait word by word. The solution was bold, and the paper’s title6 makes the point: attention is enough. Reading in order was thrown out completely. Each position turns to the others directly, all the connections are computed at the same time and spread across thousands of GPU cores. The architecture was called the transformer, and all the big models stand on it: GPT, Claude, Gemini, Llama, DeepSeek. The T in GPT stands for Transformer.
You can see how the exchange itself works on our familiar question:
Which camera should I buy for a blog?
By the end of the sentence, a lot is already known: it’s about a camera, someone wants to buy it, it’s needed for a blog. But this information sits in different positions. For each position, three new vectors are computed from its vector. The first describes what this position is looking for. The second describes how it can be useful to others. The third is what it will pass on if it gets picked. They’re called query, key and value. One position’s query is compared with the keys of the others using the same old “multiply and add,” and the scores are turned into shares: take more from here, less from there. Then the values are mixed in those shares:
new “blog” vector = 0.6 × information from “camera”
+ 0.3 × information from “buy”
+ 0.1 × information from the rest
The shares here are for illustration, but this is exactly how the operation works. “Multiply and add” again, except that now the coefficients aren’t baked into a matrix in advance but depend on the text itself.
One such mix isn’t enough: a sentence has, all at once, the link between an action and its object, qualifiers, negations, parts of one name. So in every layer several attention mechanisms work in parallel. They’re called heads, and each one learns its own connections. Nobody assigned one head to grammar and another to camera names.
After the exchange, each vector goes through the layer from Chapter 1: a matrix plus nonlinearity. Attention gathered the information, the layer processed it. Then attention again, a layer again, and so on dozens of times in a row. So that nothing gets lost at that depth, the result of each step is added to what was already there (these are called residual connections7), and the scale of the numbers is normalized8.
Two agreements remain. The first is about order. In “The dog chased the cat” and “The cat chased the dog” the words are the same, but the cat’s fate is different. Since reading in order was thrown out, order has to be supplied separately: a marker of each token’s place in the sentence is mixed into its vector. The second is about time. In GPT-family models9, each position sees only itself and what comes before it. The future is closed to it. So the early “camera” in our question knows nothing about “blog,” but “blog” knows about the camera, and the last position knows about everything. When the model writes an answer, this rule holds by itself: the future words simply don’t exist yet.
What comes out the other end? After dozens of layers, the vector at the end of the question is no longer a dictionary vector. It holds “a camera for a blog that someone is about to buy.” That’s also how the two mice part ways: one has “cat” among its neighbors, the other “laptop,” and after the very first layers their vectors are different. A name made of several tokens comes together the same way: a later piece takes the earlier ones into account, and the following positions get the whole name. There’s no need to glue it back into one row of the vocabulary.
In 2024 Anthropic showed that these internal vectors are organized meaningfully. Among the numbers Claude computes on the way to an answer, they found a direction associated with the Golden Gate Bridge and artificially amplified it. The result was Golden Gate Claude10: the model steered toward the bridge in any conversation, even one about a pie recipe. The directions from the previous section live not only in the table at the input but also deep inside the computation. They can be found, and they can be tweaked.
The whole mechanism is explained in pictures by 3Blue1Brown: on the transformer11 and on attention12.
1997 Hochreiter and Schmidhuber: LSTM – the network reads in order, but with memory
2014 Bahdanau and colleagues: attention – the translator looks back at the original
2016 Google Translate: translation switches to a neural network with attention
2017 Vaswani and colleagues: the transformer – reading in order is thrown out,
attention stays, and it spreads across GPUs
2024 Golden Gate Claude: a direction inside the model is found and amplified
Now the model has a vector that holds the whole question. But it’s still numbers, and the screen needs text. How does the next word of the answer come out of a vector?
Sources
-
Hochreiter and Schmidhuber, “Long Short-Term Memory,” 1997. A network with controlled memory that carried almost all neural network work with text until 2017. — https://www.jku.at/institut-fuer-machine-learning/forschung/publikationen/older/ ↩
-
Cho et al., “On the Properties of Neural Machine Translation: Encoder–Decoder Approaches,” 2014. Translation quality through a single vector drops as sentences get longer. — https://arxiv.org/abs/1409.1259 ↩
-
Bahdanau, Cho, and Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” 2014, ICLR 2015. Attention as an add-on to a sequential translator. — https://arxiv.org/abs/1409.0473 ↩
-
Wu et al., “Google’s Neural Machine Translation System,” 2016. Google Translate switches to a neural network with attention. — https://arxiv.org/abs/1609.08144 ↩
-
Vaswani et al., “Attention Is All You Need,” 2017. The paper that invented the transformer. — https://arxiv.org/abs/1706.03762 ↩
-
Wired, “8 Google Employees Invented Modern AI. Here’s the Inside Story,” 2024. How the transformer paper was written and where its title came from. — https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/ ↩
-
He et al., “Deep Residual Learning for Image Recognition,” 2015, CVPR 2016. Residual connections for training deep networks. — https://arxiv.org/abs/1512.03385 ↩
-
Ba, Kiros, and Hinton, “Layer Normalization,” 2016. Normalizing the scale of numbers inside the network. — https://arxiv.org/abs/1607.06450 ↩
-
Radford et al., “Language Models are Unsupervised Multitask Learners” (GPT-2), 2019. A model in which each position sees only the preceding text. — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf ↩
-
Anthropic, “Golden Gate Claude,” 2024. A “Golden Gate Bridge” direction was found inside the model and amplified, and the model started talking about the bridge in every answer. — https://www.anthropic.com/news/golden-gate-claude ↩
-
3Blue1Brown, “Transformers, explained visually.” The transformer in pictures. — https://www.youtube.com/watch?v=wjZofJX0v4M ↩
-
3Blue1Brown, “Attention in transformers, step-by-step.” The attention mechanism in pictures. — https://www.youtube.com/watch?v=eMlx5fFNoYc ↩