Part I · Chapter 2 · First there was a token · Section 3 of 4

When the vocabulary became a limit

Maksim Gurchenkov

BPE shortened sequences and made it possible to assemble rare words from familiar parts. But the vocabulary has to be prepared in advance. It has already decided which combinations get their own number and which stay a bundle of pieces. A convenient way of packing text becomes one more condition the model has to adapt to.

Can we skip the packing? We already have bytes, there are only 256 of them, and any UTF-8 text is made of them. Why merge them into a big vocabulary first if the neural network will have to learn the relationships between the parts anyway?

In 2021 Linting Xue and colleagues at Google1 introduced ByT5. They wanted to get rid of complicated preliminary cutting of text and showed that a transformer can, with small changes, be trained to work directly with bytes. In their experiments this approach was competitive, coped better with noise in the text and helped on tasks sensitive to how words are spelled.

This is different from byte-level BPE in GPT-2. There, bytes are first merged into larger units. Here, the bytes themselves go into the model:

byte-level BPE:  text → UTF-8 bytes → merged into tokens → model
byte model:      text → UTF-8 bytes → model

The big vocabulary was gone, but the old length problem came back. Instead of one large fragment, the model has to process several small ones. The same text takes more steps. So on top of the question of quality comes the question of cost: how much computation will it take to train such a model and get an answer out of it?

Meanwhile the vocabulary turned out to have another problem. In 2023 Jessica Rumbelow and Matthew Watkins2 described unusual failures of GPT models on specific tokens. One of them corresponded to a string with the username SolidGoldMagikarp. The models could fail even when asked simply to repeat it. One of the explanations discussed was a mismatch between the data used to build the vocabulary and the data used to train the model itself. The token found a place in the vocabulary, but useful work with it never came together. Having its own number doesn’t mean the model knows well what’s behind it.

In December 2024 Artidoro Pagnoni and colleagues at Meta and the University of Washington3 proposed the Byte Latent Transformer, BLT. They took on the computational cost of long byte sequences. Bytes were grouped into patches of variable length, and the main part of the model worked with representations of those patches. The boundaries were chosen by how hard it is to predict what comes next: predictable stretches can be grouped into bigger chunks, difficult ones broken down in more detail.

Merging was back, but it worked differently now. BPE looks up suitable fragments in a vocabulary prepared in advance. BLT has no such list of required pieces: the grouping depends on how predictable the current text is. The result was a way to spread computation unevenly and spend more work where it’s needed more. Under the conditions studied, this made it possible to combine quality with more economical processing.

In September 2026 Kalyani Marathe, Pagnoni and colleagues4 set the next goal: teach a small byte model from a large token model. This kind of training is called distillation. The teacher reports which continuations it considers likely, and the student learns to reproduce its estimates. But the teacher scores large tokens, and the student scores individual bytes. They needed a way to convert token probabilities into byte probabilities.

In the End-Of-Token variant, the authors added a special marker at the end of each of the teacher’s tokens. Now the conversion knew where each of the teacher’s pieces ended, and its probabilities could be preserved exactly. The small byte vocabulary stayed, but the boundaries of the teacher’s tokens were still present in training.

On a small compute budget, token models were ahead. Byte models started out worse but kept improving as spending grew. The advantage of up to four percentage points in average test accuracy was a forecast: it came from extending the measured curves beyond the experiments actually run. Generation with byte models remained expensive. The trade-off shifted, but it didn’t go away.

2021  Xue and colleagues:       ByT5 processes bytes without a vocabulary of word pieces
2023  Rumbelow, Watkins:        a token in the vocabulary doesn't guarantee the model handles it well
2024  Pagnoni and colleagues:   BLT groups bytes by how predictable the text is
2026  Marathe and colleagues:   a token model teaches a byte model by converting its estimates

Even a “tokenless” model still has discrete input units: they can be the bytes themselves. What changes is the size of the pieces and how they’re prepared. So what can you actually learn about a model’s familiarity with a company by looking at how many pieces it cut the company’s name into?

Sources

  1. Xue et al., “ByT5: Towards a Token-Free Future with Pre-Trained Byte-to-Byte Models,” preprint 2021, TACL 2022. A transformer working directly with bytes, and its computational trade-offs. — https://arxiv.org/abs/2105.13626 ↩

  2. Rumbelow and Watkins, “SolidGoldMagikarp (plus, Prompt Generation),” 2023. Observations of unusual failures on specific tokens; the possible explanations shouldn’t be extended to all rare names. — https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation ↩

  3. Pagnoni et al., “Byte Latent Transformer: Patches Scale Better Than Tokens,” 2024. Dynamic grouping of bytes and allocation of computation. — https://arxiv.org/abs/2412.09871 ↩

  4. Marathe et al., “Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models,” 2026. Distillation into byte models, measured relationships and scaling forecasts. — https://arxiv.org/abs/2609.12303 ↩

Get chapter 4 when it’s out