Part I · Chapter 1 · The very basics · Section 2 of 3

Matrices and neurons

Maksim Gurchenkov

A computer executes programs, but every program is written by a human: rule by rule, step by step. And it quickly turned out that there are tasks for which writing out all the possible rules by hand is next to impossible. Try to put into rules what makes a cat in a photo different from a dog. Or what the phrase “what camera should I buy for my blog” means. Nobody managed it.

The tool that eventually solved these tasks is called a matrix.

Matrices themselves were invented long before any AI. They’re a tool from a branch of math called linear algebra: Arthur Cayley described matrix algebra back in 18581, a hundred years before the first neural networks. At the time it was a pure game of the mind, an abstraction with no application. Then matrices were used to describe quantum mechanics, then computer graphics, and now intelligence. This happens in math all the time: a tool appears long before the problem for which it turns out to be essential.

In general, a matrix is just an ordered set of numbers that can be pictured as a table. For example, this one:

M = [ 2.0  0.5 ]
    [ 0.6  1.8 ]

In conversations about AI you’ll also often hear the word “vector.” Well, a vector is a matrix too, just with only one row or one column, depending on how it’s written in a particular formula; the orientation is fixed and affects the result of the operation. For example, here is something that is both a 1×2 matrix and simply a vector of two values:

V = [ 1.0  -0.2 ]

The bridge from matrices to intelligence was built in 1943, and it started not with math, but with biology. A nerve cell in the brain, a neuron, works roughly like this: branches from thousands of neighboring cells reach it, and a signal arrives along each one. The connections aren’t equal: some are strong, and the signal comes through them “loudly,” others are weak, barely audible. The cell gathers everything that has come in, and if the total signal is strong enough, it “fires” and sends a signal onward, to its own neighbors.

McCulloch and Pitts2 wrote this behavior down in numbers, and the result was the mathematical neuron, the simplest model of a nerve cell. The signal from each neighbor is an input, just a number. The strength of the connection is a weight, another number. Each input is multiplied by its weight, the products are added up, and the sum is compared with a threshold: if it reaches it, the neuron sends a signal.

inputs:   x₁, x₂, x₃          signals from the neighbors
weights:  w₁, w₂, w₃          the strength of each connection

x₁·w₁ + x₂·w₂ + x₃·w₃  ≥  threshold   →   the neuron “fires”

All the work of a nerve cell came down to one action: “multiply and add.” And McCulloch and Pitts were inspired by that same Boolean logic: their paper shows directly how the same three key operations, AND, OR and NOT, are built from such neurons.

So what do matrices have to do with it? Take a close look at multiplying a vector by a matrix: to get each number of the result, you multiply a row of numbers and add up the products. That’s literally the same operation a neuron does. One neuron is one “multiply and add.” And neurons don’t work alone, they work in layers: several neurons look at the same inputs, but each has its own weights. Let’s write the inputs as a vector V, and the weights of all the neurons in the layer as a matrix M, one column per neuron. Then the whole layer is computed with a single multiplication V × M, and the output is a new vector V': one number from each neuron. The operation itself is as simple as it gets, it’s taught in schools in many countries, and it’s not much harder than multiplying two numbers.

V × M = [ 1.0  -0.2 ] × [ 2.0  0.5 ]  =  [ 1.88  0.14 ] = V'
                        [ 0.6  1.8 ]

first neuron, its weights 2.0 and 0.6:    1.0·2.0 + (-0.2)·0.6  =  1.88
second neuron, its weights 0.5 and 1.8:   1.0·0.5 + (-0.2)·1.8  =  0.14

A whole layer of neurons as one vector-matrix multiplication

This operation is what all modern neural networks, including ChatGPT, Claude and Gemini, run on at their core.

The dream of a machine that handles tasks whose rules can’t be written down on its own was called artificial intelligence: the term AI was proposed by John McCarthy in 19553. Many approaches to this dream were tried, but in the end one won: machine learning. Its idea turns programming upside down: it’s not a human who writes the rules, the machine learns them itself from examples.

The main tool of this approach today is neural networks. And a neural network trained on text is called an LLM (Large Language Model). ChatGPT, Claude, Gemini and the other heroes of this book are built around LLMs.

AI                 the dream: a machine solves tasks whose rules can't be written down
 ML                the method: rules aren't written by a human, they're learned from examples
  Neural networks  the tool: layers of “multiply and add”
   LLM             a neural network trained on text   ←   what this whole book is about

And just as computers are “just” zeros and ones, with operations and conventions, neural networks under the hood are “just” sets of numbers arranged into so-called matrices, which you can do math operations on and get matrices out again! And on top of that, our conventions about what it all means.

In 1958, Rosenblatt4 turned the mathematical neuron into real hardware. His machine was called the perceptron, and inside it was the same familiar “multiply and add” operation: the inputs are multiplied by the weights, the products are added up, and the sum is compared with a threshold. If it reaches the threshold, the machine answers “yes”; if it doesn’t, “no.”

What was new was something else: the perceptron was the first in history to learn from examples. The learning is simple. The machine is shown an example, it answers, and the answer is checked against the correct one:

answer matches the correct one   →  leave the weights alone
answer is wrong                  →  nudge each weight a little
                                    toward the correct answer

And so on thousands of times, example after example. Nobody wrote the rules, the weights tuned themselves. It was a sensation: newspapers promised machines that would soon walk, talk and be aware of themselves, and the military handed out funding. It seemed a thinking machine was within arm’s reach.

But in 1969, Minsky and Papert5 rigorously proved that a single “multiply and add” operation is only enough for the simplest tasks. Here’s the thing. A perceptron’s answer always works the same way: the bigger the sum, the closer to “yes.” But there are tasks where the correct answer works differently. For example: two inputs, and the answer should be “yes” only when exactly one of them is on.

only A is on     →   the sum reaches the threshold   →   “yes”   correct
only B is on     →   the sum reaches the threshold   →   “yes”   correct
A and B are on   →   the sum gets even bigger        →   “yes”   but it should be “no”!

The second input can only increase the sum, there’s nothing to turn it back. And however you shift the weights, you can’t fix it: the problem isn’t in the weights, it’s in the operation itself. “Multiply and add” is one step, and some tasks can’t be solved in one step. They promised a thinking machine, and it tripped over a little puzzle with two inputs. The enthusiasm collapsed, the funding followed, and the “AI winter” set in.

Cayley:               a matrix is a table of numbers, invented a century before neural networks
                         ↓
McCulloch and Pitts:  a neuron is “multiply and add” with a threshold,
                      and a layer of neurons is multiplication by a matrix
                         ↓
McCarthy:             the dream of a machine solving such tasks was named AI
                         ↓
Rosenblatt:           weights don't have to be set, they can be learned from examples
                         ↓
Minsky and Papert:    but one layer hits a ceiling → the “AI winter”

The history of artificial intelligence could have ended right there. Luckily, it didn’t.

One linear layer doesn’t untangle the pattern; nonlinearity and depth separate it

Sources

  1. Arthur Cayley, “A Memoir on the Theory of Matrices,” 1858—the work that set out matrix algebra, a century before neural networks. — https://en.wikisource.org/wiki/A_Memoir_on_the_Theory_of_Matrices ↩

  2. McCulloch, Pitts, “A Logical Calculus of the Ideas Immanent in Nervous Activity,” 1943—the first mathematical neuron, built from Boolean logic. — https://www.cs.cmu.edu/~./epxing/Class/10715/reading/McCulloch.and.Pitts.pdf ↩

  3. McCarthy, Minsky, Rochester, Shannon, “A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence,” 1955—the document in which the term artificial intelligence first appeared. — http://jmc.stanford.edu/articles/dartmouth/dartmouth.pdf ↩

  4. Rosenblatt, “The Perceptron: A Probabilistic Model,” 1958—the first learning machine: the weights are adjusted based on errors. — https://homepages.math.uic.edu/~lreyzin/papers/rosenblatt58.pdf ↩

  5. Minsky, Papert, “Perceptrons,” 1969—proof of the limits of a single-layer network, which started the “AI winter.” — https://en.wikipedia.org/wiki/Perceptrons_(book) ↩

Get chapter 4 when it’s out