Part I · Chapter 2 · First there was a token · Section 1 of 4

From the telegraph to ASCII

Maksim Gurchenkov

Any modern computer is “just” zeros and ones, and any LLM is “just” matrices with nonlinearity between them. On top of all that sit simple math operations and our agreements about how to interpret the whole thing. Since we’ve already covered the zeros, ones, matrices and operations, this chapter and the next are about those agreements.

Neither zeros and ones nor matrices mean anything on their own. A record a machine can read appeared the moment we agreed on how to encode and decode text. And people spent almost two hundred years getting to that agreement on how to encode words as numbers. It didn’t start with AI, or even with computers. It started with the telegraph.

The word TOKEN built from paper fragments and numbers

A telegraph is a wire that can carry only one thing: current or no current. Press the key and something clicks at the other end of the wire. Let go and there’s silence. The same zero and one from Chapter 1, just a hundred years before Shannon. You can’t send letters down a wire like that, only a sequence of clicks. So in 1838 Morse and Vail1 worked out how to write every letter in clicks: a short click is a dot, a long one is a dash. That’s how Morse code2 was born. But which letter gets which pattern? They found the answer in a print shop. Back then text was set by hand from small metal letters called type, and the type for each letter sat in its own compartment of a case. Common letters got more pieces cast and a bigger compartment, rare ones a tiny one. One look at a case like that was enough. A common letter got a short code: E is a single dot. A rare one got a long code: Q is dash-dash-dot-dash. That’s how encoding picked up the idea this whole section stands on: encode the common short and the rare long, and the message gets shorter on average.

Morse code had one flaw: only a human could receive it. Dots, dashes and pauses of different lengths between them, and the operator told by ear, by rhythm, where one letter ended and the next began. As long as a person sat at the receiving end, nobody minded. But by the 1870s telegraph lines had become a bottleneck: a wire between cities was expensive, and it carried only one conversation at a time.

In 1874 Émile Baudot3 figured out how to run several at once over a single wire. Two disks spin at the same speed at both ends, each turn is split into equal slices, the first slice belongs to the first operator, the second to the second, and so on around the circle. For this to work, each slice has to hold exactly one letter, which means all letters have to be the same length. With Morse code, where E is more than ten times shorter than Q, you can’t build that. So Baudot gave every letter a code of exactly five signals: current or no current, five times in a row. Morse’s savings were gone, but the receiver no longer needed an ear. It counts five signals, finds them in a 32-row table and prints the letter by itself, no human required. A letter became a row number in a table.

Seventy years later, in 1948, Claude Shannon4 (the same one from Chapter 1 who noticed in 1937 that 0 and 1 themselves, and all three operations, could be built from electrical circuits) showed that the amount of information in a message can be measured, and that the predictable compresses: the better you can guess what comes next, the fewer bits you need to send it. And in a 1951 paper5 he ran an experiment that today reads like a description of a “Human GPT.” He had people guess a text letter by letter and used their accuracy to measure how predictable English is. The key idea: simply predicting the next bit of text is a way to grasp the whole language. Shannon did this with people, seventy years before language models. A year later MIT student David Huffman6 settled the question Morse had answered by eye. He came up with an algorithm that builds the best frequency code for any data. His code still runs inside every ZIP file.

Baudot’s table survived into the computer age, and there the story repeated on a bigger scale. Inside a computer a letter is also stored as a number in a table, except that by the late 1950s every manufacturer had its own table. IBM alone had several incompatible ones, and two machines in neighboring departments couldn’t exchange text. Bob Bemer7 of IBM recalled that what finally did him in was the thought of text editors: they needed both uppercase and lowercase letters, and those didn’t fit into the old 64 characters. The solution turned out to be human, not technical: get a committee together and agree. In 1963 the ASCII table8 came out: one for everyone, 128 characters, each with its own number. Bemer has been called the father of ASCII ever since, and we also owe him the Escape key, the backslash and curly braces. It’s an agreement in the most literal sense of the word: people from different companies sat down at a table and settled which number means which letter. And the agreement decides everything. The same byte 01000001 is the letter “A” in ASCII, the number 65 in Leibniz’s system and a shade of gray if you read it as a pixel. The bits themselves have no meaning. Meaning lives in the interpretation.

1838  Morse and Vail:   common letters get short codes
1874  Baudot:           every letter gets a code of the same length,
                        a letter = a row number in a table
1948  Shannon:          the predictable compresses; language can be measured
                        by guessing the next letter
1952  Huffman:          the best frequency code
1963  Bemer, ASCII:     one “number = letter” table for everyone

From Morse code and the Baudot code to ASCII

So by 1963 we had learned to store text in a machine: a letter equals a number, and there’s one table for everyone. Meanwhile the Morse line, where the common gets a short code, went its own way.

Sources

  1. Alfred Vail, co-author of Morse code; he is the one credited with counting the letters in a printer’s type case. Wikipedia. — https://en.wikipedia.org/wiki/Alfred_Vail ↩

  2. Morse code, the first frequency-based encoding of text, 1838. Wikipedia. — https://en.wikipedia.org/wiki/Morse_code ↩

  3. Émile Baudot, the Baudot code, 1874. The first fixed-length code: a letter = five units, and text became readable by a machine. Wikipedia. — https://en.wikipedia.org/wiki/Baudot_code ↩

  4. Claude Shannon, “A Mathematical Theory of Communication,” 1948. The paper that gave birth to information theory. — https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf ↩

  5. Claude Shannon, “Prediction and Entropy of Printed English,” 1951. The predictability of language, measured by guessing the next letter. — https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf ↩

  6. David Huffman, “A Method for the Construction of Minimum-Redundancy Codes,” 1952. An optimal frequency code and a direct descendant of Morse’s idea; together with LZ77 it works inside every ZIP file. Wikipedia. — https://en.wikipedia.org/wiki/Huffman_coding ↩

  7. Bob Bemer, the “father of ASCII,” an IBM engineer who pushed through a single character table. Wikipedia. — https://en.wikipedia.org/wiki/Bob_Bemer ↩

  8. ASCII, the “byte = character” agreement adopted by a committee in 1963. Wikipedia. — https://en.wikipedia.org/wiki/ASCII ↩

Get chapter 4 when it’s out