Part I · Chapter 3 · Where meaning lives · Section 1 of 3

Words get coordinates

Maksim Gurchenkov

A token ID is a row number in a vocabulary, and nothing more. 9427 is greater than 9426, but “camera” isn’t greater or smaller than “coffee.” The numbers can’t tell you that coffee is closer to tea than to a camera. For meaning to show up in numbers, we need one more agreement. And, just like tokens, it was found long before neural networks and not for them at all.

Linguists in the 1950s had a problem of their own: they wanted an exact science of language, and “meaning” didn’t fit into it. You can’t measure it or test it, only describe it in words through other words. In 1954 Zellig Harris1 proposed going around meaning: don’t ask what a word means, look at where it occurs. After which words, before which, in which constructions. Meaning is invisible, but usage can be collected and compared. In 1957 the British linguist John Firth2 packed the same thought into a phrase that has been quoted in every other paper on language ever since: “You shall know a word by the company it keeps.” In other words, you can tell a word by the crowd it hangs out with.

In the sentences “I ordered a cup of coffee” and “I ordered a cup of tea,” only one word changes. Both things get ordered, poured, drunk and offered to guests. Without knowing what tea is yet, you could guess from these neighbors alone that it belongs to the same circle of things as coffee. A “camera,” on the other hand, gets bought, charged and set up, but never poured into a cup. The method has a limit: “hot” and “cold” also stand in the same places and mean opposite things. But there was now a foothold: you can judge a word’s meaning from observable data, without opening a dictionary.

The next step was taken not by linguists but by search engines. In 1975 Gerard Salton and colleagues3 were building a document search system at Cornell and ran into the fact that matching words doesn’t answer the main question: which document is closer to the query? They wrote each document as a vector from Chapter 1: one number for each word in the vocabulary. In the simplest case, that’s the number of times the word appears in the text.

Let’s take a world with only three words, and three documents:

                       coffee    tea    camera
drinks text               4       3        0
another drinks text       8       6        0
photography text          0       0        5

Each row is a vector, and a vector can be drawn as an arrow. The first and second arrows point the same way: their ratio of coffee to tea is the same, the second text is just twice as long. The third points somewhere else entirely. Salton proposed measuring how close texts are by the direction of their arrows, and it’s computed with the familiar “multiply and add”:

first and second:   4·8 + 3·6 + 0·0 = 50
arrow lengths:      5 and 10
50 / (5 × 10) = 1          →   directions match

first and third:    4·0 + 3·0 + 0·5 = 0
0 / (5 × 5) = 0            →   nothing in common

This number is called cosine similarity. Dividing by the lengths is needed so that a long article about tea doesn’t end up closer to the query than a short, precise one just because of its size. That’s how a space appeared where closeness in topic is closeness in direction. Long before neural networks moved into it.

Salton’s vectors still had a hole: an article about a “car” wouldn’t come up for the word “automobile.” Same meaning, different columns in the table. In 1990 Scott Deerwester, Susan Dumais and their colleagues4 took a big “words × documents” table, the very matrix from Chapter 1, and mathematically compressed it down to about a hundred hidden axes. Words that live in the same surroundings ended up next to each other on those axes: one author writes “car,” another “automobile,” but both write about engines, roads and fuel. Nobody wrote out pairs of synonyms. For the first time, the connection between words was computed from text statistics, exactly as Harris had suggested.

Neural networks entered this space in 2003, and once again the reason was practical. Language models of that time counted how often chains of two or three words in a row appeared in texts. About a chain that never appeared in the texts, such a model could say almost nothing: it had never seen “a cup of cocoa,” so people must not say that. Yoshua Bengio and colleagues5 proposed learning a vector for each word right during training on prediction, so that similar words could share experience: if “coffee” and “cocoa” have close vectors, what was learned about a cup of coffee carries over to a cup of cocoa. In Deerwester’s approach, vectors came out of one table all at once. In Bengio’s, they’re tuned by errors, example after example, like the weights of a perceptron. It worked, but it learned painfully slowly.

In 2013 Tomáš Mikolov and colleagues at Google6 threw out of Bengio’s network everything it could do without, and the resulting word2vec learned on billions of words instead of millions. The task stayed Harris’s: guess a word from its surroundings, or the surroundings from a word. And the result was unexpected. It turned out you could do arithmetic with the vectors:

king − man + woman ≈ queen

Take the vector for “king,” subtract “man,” add “woman,” and the closest thing to the result in the table is “queen.” Nobody explained monarchy or gender to the network. The relationships between meanings showed up on their own, from nothing but the prediction task. The trick doesn’t work with every set of four words, but the very possibility is striking: meaning can be added and subtracted.

In an LLM all of this is gathered into one table, and it’s called the embedding table. An embedding is the vector that represents a token for computation. The table has as many rows as there are tokens in the vocabulary, and each row holds several thousand numbers. The token ID is the row number: the first thing the model does with ID 9427 is take row 9427, and from then on it works not with a number but with the vector for “camera.”

9427 → row 9427 → [ 0.3, -0.8, 0.1, 0.6, ... ]

The numbers here are made up. Nobody filled the table in by hand: it’s learned together with the other matrices, the same way as with Bengio and Mikolov. And the coordinates have no labels. There’s no separate number for “drink” or “expensive”: one feature is smeared across many coordinates, and one coordinate takes part in many features.

You can see a space like this in the Embedding Projector7: a cloud of thousands of word vectors you can spin with your mouse. The picture flattens hundreds of coordinates down to three and loses some of the relationships, but the main thing is visible: the words that end up next to each other are ones with similar meanings, not similar letters or neighboring numbers.

A company name works the same way as “coffee”: its vectors are learned from the surroundings it appeared in. But there’s no single arrow that holds everything about the company. A name made of several tokens is several rows of the table, and the starting vectors are only the beginning of the computation.

1954  Harris:                     words can be compared by their surroundings
1957  Firth:                      “you shall know a word by the company it keeps”
1975  Salton:                     texts are vectors, closeness is the direction of arrows
1990  Deerwester and colleagues:  connections between words computed from text statistics
2003  Bengio:                     a neural network learns a vector for every word
2013  Mikolov:                    word2vec – the same on billions of words,
                                  and you can do arithmetic with meaning
      LLM:                        the embedding table, token ID = row number

But the word “mouse” has a single row in the table. And a mouse next to a cat and a mouse next to a laptop are different mice. How does one and the same vector figure out which one we mean?

Sources

  1. Zellig Harris, “Distributional Structure,” 1954. Studying language through the distribution of its elements. — https://corpus.bfsu.edu.cn/Zellig_Harris_Distributional_Structure_Word_1954.pdf ↩

  2. John Rupert Firth, “You shall know a word by the company it keeps,” 1957. Wikipedia. — https://en.wikipedia.org/wiki/John_Rupert_Firth ↩

  3. Salton, Wong, and Yang, “A Vector Space Model for Automatic Indexing,” 1975. Texts as vectors and how to compare them. — https://doi.org/10.1145/361219.361220 ↩

  4. Deerwester et al., “Indexing by Latent Semantic Analysis,” 1990. Hidden structure from a table of words and documents. — https://www.microsoft.com/en-us/research/publication/indexing-by-latent-semantic-analysis/ ↩

  5. Bengio et al., “A Neural Probabilistic Language Model,” 2003. Word vectors learned together with predicting what comes next. — https://www.jmlr.org/papers/v3/bengio03a.html ↩

  6. Mikolov et al., “Efficient Estimation of Word Representations in Vector Space,” 2013. word2vec. — https://arxiv.org/abs/1301.3781 ↩

  7. Embedding Projector, an interactive cloud of word vectors. — https://projector.tensorflow.org/ ↩

Get chapter 4 when it’s out