After that, compression tools turned “common means short” into an automatic routine. In 1977 Lempel and Ziv1 proposed compressing data with a dictionary: put frequently repeated sequences into a dictionary and replace them in the text with a short reference. Every ZIP file on your drive runs on descendants of this idea.
Meanwhile the line of character tables hit its own ceiling. Bemer’s table has 128 characters: English letters, digits, punctuation and control codes. There’s no room in it for other writing systems. Every country set up its own table on top of ASCII: Shift JIS in Japan, KOI-8 and later Windows-1251 in Russia, each with its own numbers for its own letters. Text written in one table and opened in another turned into garbage made of random characters. In Japanese the phenomenon even got a name, 文字化け, “mojibake,” and in the 1980s everyone who sent a file abroad knew it. In 1988 Joe Becker of Xerox2 proposed making one table for all the world’s writing systems at once, and Unicode came out in 1991. It was expanded step by step, adding characters from different scripts and later emoji. One question remained: how to write the numbers from such a table as bytes without breaking old ASCII files. In 1992 Ken Thompson and Rob Pike3 sketched UTF-8 on a napkin over dinner at a diner in New Jersey. An English letter stays one byte, as with Bemer, a Russian letter takes two bytes, and many ideographs take three. Almost the entire internet is stored in this encoding today, and it’s also what sits at the input of language models.
And in 1994 Philip Gage4 described a very simple kind of dictionary compression, byte pair encoding, or BPE for short: find the most common pair of neighboring symbols in the data, merge it into a new symbol and add it to the dictionary. Repeat until the dictionary is full. On a tiny corpus of the words low, lower, newest and widest, the first steps look like this:
step 1: most common pair e + s (9 times) → new symbol es
step 2: most common pair es + t (9 times) → new symbol est
step 3: most common pair est + _ (9 times) → new symbol est_
step 4: most common pair l + o (7 times) → new symbol lo
The suffix “-est” and the root “lo” grow out of the letters by themselves, without a single grammar rule. The algorithm doesn’t care that this is language. It sees only frequencies.

All of this, from Morse to Gage, was done to save something: telegraph money, disk space. Nobody was thinking about intelligence, and it was exactly the same with matrices: the tool appeared long before its main job.
The bridge to neural networks was built in 2016, and the reason was purely practical. Neural machine translation systems had to decide what pieces to feed text in, and each of the two obvious options had its own price:
by character: the vocabulary is small, but sequences get longer –
the model has more units to process
by word: you can't list every word in advance: names, typos, new words...
a word outside the vocabulary = “who knows what”
Sennrich and colleagues5 took BPE, a twenty-year-old compression algorithm, and found the sweet spot: let the vocabulary build itself from the data. Common words can become whole tokens, and rare ones are assembled from familiar pieces, down to individual characters. Now a new word doesn’t necessarily need a new row in the vocabulary.
The result was a table like Baudot’s and Bemer’s, with its own number for each entry. Only the size of this table is chosen by the developers: tens or hundreds of thousands of rows. And the fragments themselves are picked by Morse’s rule and Gage’s recipe: the more often a sequence appears in the texts prepared for building the vocabulary, the better its chances of becoming a separate token with its own number.
One more difficulty remained: many tools first split text into words by spaces. But Japanese text, for example, doesn’t work that way. In 2018 Taku Kudo and John Richardson of Google6 introduced SentencePiece, which learns how to cut text directly from the text itself, without splitting it into words first. BPE became one of the methods it supports. The task can be solved with different algorithms, even when all of them produce tokens in the end.
In 2019, for GPT-27, OpenAI started merging UTF-8 bytes instead of letters. That way the vocabulary is guaranteed to cover any text in any language, since there are only 256 bytes. The starting length of the record still differs, though: an English letter starts as one byte, a Russian one as two, many ideographs as three. How many tokens you end up with depends both on that length and on the merges the tokenizer has learned.
Let’s take our sentence and the o200k_base vocabulary:
"Which camera should I buy for a blog?"
↓
[ 34405, 9427, 1757, 357, 3877, 395, 261, 4109, 30 ]
The fragment “ for,” together with the space before the word, is one whole token here, number 395. The fragment “ camera” got its own number too, 9427. A word that isn’t in the vocabulary as a whole, such as the name of a new brand, can be assembled from several pieces.
You can see this at tiktokenizer.vercel.app8: any text you paste there crumbles into colored pieces, and the whole mechanism becomes clear in a minute. And if you want to take a tokenizer apart down to the last screw, you can do it with Karpathy, who builds one from scratch in two hours in “Let’s build the GPT Tokenizer”9.
The token agreement didn’t stop at text. A transformer is a type of neural network that works with sequences. An image first has to be turned into such a sequence. In 2020 Dosovitskiy and colleagues at Google10 proposed a solution in a paper with a title that says it all, “An Image is Worth 16×16 Words”: the image is cut into small squares, say 16 by 16 pixels, each square gets a set of numbers and then goes into the transformer. Images can also be handled as a sequence of numeric representations, but the rules for cutting up text don’t carry over to them literally.

1977 Lempel and Ziv: common sequences go into a dictionary
1991 Becker, Unicode: one table for all the world's writing systems
1992 Thompson and Pike: UTF-8 – character numbers are written as bytes
1994 Gage: BPE – compression by merging the most common pairs
2016 Sennrich: BPE comes to neural networks: a table of numbers
like Baudot's and Bemer's, built by frequency like Morse's
2018 Kudo, Richardson: SentencePiece learns to cut text without a ready-made word split
2019 GPT-2: UTF-8 bytes are merged, the vocabulary covers any language
2020 Dosovitskiy: an image is cut into squares, and those are tokens too
Sources
-
Ziv and Lempel, “A Universal Algorithm for Sequential Data Compression” (LZ77), 1977. Dictionary compression, the ancestor of ZIP. Wikipedia. — https://en.wikipedia.org/wiki/LZ77_and_LZ78 ↩
-
Unicode, a single character table for all writing systems: proposed by Joe Becker in 1988, first version released in 1991. Wikipedia. — https://en.wikipedia.org/wiki/Unicode ↩
-
Rob Pike, “UTF-8 history,” 2003. How Ken Thompson and Rob Pike invented UTF-8 over dinner at a diner in September 1992. — https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt ↩
-
Philip Gage, “A New Algorithm for Data Compression,” 1994. The original byte pair encoding. Wikipedia — https://en.wikipedia.org/wiki/Byte_pair_encoding ; original paper (PDF) — https://www.derczynski.com/papers/archive/BPE_Gage.pdf ↩
-
Sennrich, Haddow, and Birch, “Neural Machine Translation of Rare Words with Subword Units,” 2015 (ACL 2016). The paper that brought the BPE algorithm into language models. — https://arxiv.org/abs/1508.07909 ↩
-
Kudo and Richardson, “SentencePiece,” 2018. Learning to cut text directly from the text, without splitting it into words first. — https://arxiv.org/abs/1808.06226 ↩
-
Radford et al., “Language Models are Unsupervised Multitask Learners” (GPT-2), 2019. BPE over UTF-8 bytes: the vocabulary covers any language. — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf ↩
-
Tiktokenizer, an interactive playground: paste in text, see the tokens. — https://tiktokenizer.vercel.app/ ; the tiktoken library (OpenAI) — https://github.com/openai/tiktoken ↩
-
Andrej Karpathy, “Let’s build the GPT Tokenizer.” The GPT tokenizer built from scratch. — https://www.youtube.com/watch?v=zduSFxRajkE ; code for the video, minbpe — https://github.com/karpathy/minbpe ↩
-
Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” 2020. An image as a sequence of square tokens. — https://arxiv.org/abs/2010.11929 ↩