Part I · Chapter 2 · First there was a token · Section 4 of 4

Tokens and languages

Maksim Gurchenkov

The token is still a handy unit of account. A context window is described by the number of units it holds, the cost of calling a model is often counted per million input and output tokens, and generation speed in tokens per second. But this currency has no single denomination. One token can hold different amounts of text in different systems. Comparing speed by token count alone is like comparing reading speed by page count without looking at the font size.

For a specific model version, the vocabulary and the cutting rules are usually fixed. A new article about a company doesn’t automatically add its name to the vocabulary. But the model can put the name together from parts it already has. Not having its own number for a name doesn’t stop the model from working with it, just as a word missing from a school dictionary doesn’t stop you from writing it with familiar letters.

How text gets cut depends on the data the tokenizer was prepared on, on how it’s built and on the language itself. In studies of multilingual tokenizers1, the same meaning in different languages required noticeably different numbers of tokens. When you pay per token, that also turns into a difference in cost2, which people call the “token tax.” The meaning of the message is the same, but it eats up more of the context window and the bill can come out higher.

Here’s our sentence in three languages in the o200k_base vocabulary used by GPT-4o. Here and below, the measurements were made with the tiktoken 0.14.0 library on strings without added leading or trailing spaces:

Which camera should I buy for a blog?    37 characters    37 bytes     9 tokens
Какую камеру купить для блога?           30 characters    55 bytes     9 tokens
ブログ用にどのカメラを買うべきですか?       19 characters    57 bytes    16 tokens

The same question takes different numbers of tokens in English, Russian and Japanese

The Russian sentence matches the English one in token count here, even though it has more bytes. The Japanese one takes more tokens, even though it has fewer visible characters. This is the result of one specific measurement, not a fixed exchange rate between languages. Another text or another vocabulary may give a different ratio.

Switching to bytes doesn’t make the lengths equal either. The thirty-seven bytes of the English sentence and the fifty-seven of the Japanese one don’t go anywhere. What goes away is the influence of a pre-built vocabulary of word pieces, but the quirks of the encoding and the different lengths of the expressions themselves remain. It helps to tell these causes apart so you don’t blame the whole difference on BPE alone.

We’ve turned text into numbers: 34405, 9427… Or left it as a sequence of bytes. A number on its own still explains nothing. 9427 isn’t “greater” or “better” than 9426. It’s a row number in a vocabulary. Morse, compression tools, BPE and byte models explain how to prepare a record for computation. But where in these numbers does meaning appear?

Sources

  1. Petrov et al., “Language Model Tokenizers Introduce Unfairness Between Languages,” 2023. Differences in representation length between languages, including when switching to characters and bytes. — https://arxiv.org/abs/2305.15425 ↩

  2. Ahia et al., “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models,” 2023. How tokenization is tied to the cost of using language models. — https://arxiv.org/abs/2305.13707 ↩

Get chapter 4 when it’s out