Neural networks: how they really work

An orchestra of tiny calculators

Forget the electronic brain of the movies. A neural network is something humbler and more surprising: thousands of dead-simple computing units, connected in layers by millions (in modern models, billions) of weighted connections. As MIT explains, each node does one thing: it receives numbers over its incoming connections, multiplies them by their weights, adds everything up and compares the total against a threshold. Above the threshold, the node “fires” and passes the signal to the next layer; below it, silence.

None of these units knows anything. The intelligence — if we want to call it that — lives entirely in the weights: millions, nowadays billions, of dials deciding how much every connection counts.

Training: tuning the dials

Here’s the part that surprises everyone who first learns it: at the start, the weights are random numbers. A newborn network can do literally nothing.

Then the studying begins. It’s shown an example — a photo labeled “cat” — and the output layer’s answer is checked. Wrong? A correction algorithm nudges the weights, ever so slightly, in the direction that would have made the answer less wrong. Repeat with the second example, the thousandth, the millionth. Micro-nudge after micro-nudge, the network comes to give consistently right answers even on inputs it has never seen.

Practical example: it’s like tuning a thousand-string instrument by ear. You play a test note, hear it’s off, slightly turn the pegs that matter most, try again. After millions of tries the instrument plays in tune — though nobody ever wrote down the tuning score.

Why “deep”: the power of layers

With few layers a network learns simple relationships. The real leap comes as the layers multiply — that’s the deep in deep learning. In an image network, the early layers learn to catch edges and contrasts; the middle ones combine edges into shapes; the last ones combine shapes into objects. Nobody designs this hierarchy of abstractions: it emerges on its own from training.

The same principle, applied to text at colossal scale, produces the models behind generative AI: GPT-3 is a network of 175 billion weights. The dials have multiplied beyond counting; the principle is still 1957’s.

A rollercoaster history

Yes, 1957: neural networks are anything but new. The theory was born in 1944 with Warren McCullough and Walter Pitts; in 1957 Frank Rosenblatt unveiled the Perceptron, the first trainable network. Then an influential critique (Minsky and Papert, 1969) froze funding for years; the 1980s brought a renaissance, the 2000s another winter as other methods took over.

The definitive turning point came with GPUs, the graphics cards: built for video games, they turned out to be perfect for the billions of parallel multiplications training requires. From there: networks tens of layers deep, internet-scale data, and deep learning leaving the labs for your phone.

To place the piece in the puzzle: neural networks are the most powerful engine of machine learning, and what a network executes is still an algorithm, with all its well-defined steps. The network doesn’t know it’s intelligent. It executes, and that’s enough.