Part I · Chapter 1 · The very basics · Section 3 of 3

Nonlinearity and depth

Maksim Gurchenkov

The most annoying part is that the solution was right on the surface, and we’ve already seen it. Remember the computer: a single instruction can’t do much either, but a sequence of instructions, where each next one works with the result of the previous one, can handle anything: it computes any math and reproduces any logic. If one step hits a ceiling, you need to take many steps. Let’s apply the same idea to neurons: let the answer be built not from one “multiply and add,” but from a chain of such steps. One step is a multiplication by a matrix, so we put multiplications in a chain: multiply the vector by the first matrix, the result by the second, then by the third, and so on.

But one does not simply take a whole bunch of matrices… All the matrices are known in advance and don’t depend on what comes in as input. And anything known in advance can also be computed in advance: multiply a hundred matrices together once and get just one.

V × M₁ × M₂ × M₃  =  V × (M₁ M₂ M₃)  =  V × M_one

This simple simplification kills the whole idea: a chain of a hundred matrices works exactly like a single matrix. As mathematicians would say: Equivalent! The wall of Minsky and Papert hasn’t gone anywhere.

But now it’s clear exactly what we need: to stop the chain from collapsing. The collapse works because the whole chain consists of nothing but multiplications and additions. Let’s insert between the multiplications something that can’t be expressed through multiplication and addition, something nonlinear!

And such a nonlinear action can be ridiculously small. Today’s standard is the ReLU rule1: “if a number is negative, replace it with zero.” Look closely: this is a condition, an “if,” something “multiply and add” didn’t have at all. But a living neuron always had it: that very threshold, “fire or stay silent,” is a natural nonlinearity, and it was already there in the McCulloch and Pitts model. By lining up bare multiplications in a chain, we lost it, and now we’re just putting it back. Which numbers get zeroed out depends on the input data, and what depends on the input can’t be computed in advance. The chain stops collapsing, each step starts adding something of its own to the answer, and a hundred matrices finally become a hundred real operations. Such a chain, matrix multiplications with nonlinearities between them, is what’s called a neural network. And each of its steps, a matrix plus a nonlinearity, is called a layer.

Without nonlinearity the layers collapse into one, with it depth starts to make sense

Why so many layers? For the same reason a program needs many instructions: each next layer works with the result of the previous one, and complexity builds up. The first layers assemble words from letters, the middle ones pick out meanings, the later ones shape the answer itself. Later, mathematicians rigorously proved2 that a shallow network would need astronomically more neurons for the same work; this is shown most beautifully in the 3Blue1Brown visualizations3. One matrix is one formula. A hundred layers is already a program of a hundred sequential steps and conditions. And the difference between them is the same as between a calculator and intelligence.

The question left was how to train such a multilayer network. In 1986, Rumelhart, Hinton and Williams4 gave the answer that is still used today: the error at the output rolls backward through all the layers and corrects each matrix. The method was called backpropagation (backward propagation of errors). And depth really started to work in 2012, when the AlexNet5 network, running on gaming graphics cards, crushed everyone at an image recognition competition. From then on, layers started to be counted in dozens and hundreds: GPT-36 has 96, and Llama 3.1 405B7 already has 126. That’s actually why such networks are called “deep,” and the whole field deep learning.

The signal goes forward, the error comes back and corrects the weights

Chain of matrices:   bare multiplications collapse back into one matrix
                        ↓
Nonlinearity:        an “if” between multiplications, and the chain doesn't collapse:
                     that's how matrices become a neural network of layers
                        ↓
Hinton and others:   the error trains many layers at once (backpropagation)
                        ↓
ReLU and GPUs:       depth started to work, layers grew into the hundreds

And so, just as simple 0s and 1s are enough to make all the magic of computers, simple matrices with nonlinearity between them turn out to be enough to create the magic of Intelligence.

This is the foundation of everything, and it’s freaking awesome. But it’s only the beginning!

Sources

  1. Glorot, Bordes, Bengio, “Deep Sparse Rectifier Neural Networks,” 2011—the work that made ReLU the standard for deep networks. — https://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf ↩

  2. Telgarsky, “Benefits of Depth in Neural Networks,” 2016—a rigorous proof of the advantage of depth. — https://arxiv.org/abs/1602.04485 ↩

  3. 3Blue1Brown, “But what is a neural network?”—the best visualization of how a neural network works. — https://www.youtube.com/watch?v=aircAruvnKk ↩

  4. Rumelhart, Hinton, Williams, “Learning Representations by Back-Propagating Errors,” Nature, 1986—backpropagation, the method for training multilayer networks. — https://www.cs.toronto.edu/~hinton/absps/naturebp.pdf ↩

  5. Krizhevsky, Sutskever, Hinton, “ImageNet Classification with Deep CNNs” (AlexNet), 2012—the moment deep learning started working on GPUs. — https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf ↩

  6. Brown et al., “Language Models are Few-Shot Learners” (the GPT-3 paper), 2020. — https://arxiv.org/abs/2005.14165 ↩

  7. Llama Team (Meta), “The Llama 3 Herd of Models,” 2024. — https://arxiv.org/abs/2407.21783 ↩

Get chapter 4 when it’s out