Neural Network

A neural network (or artificial neural network) is the basic building block of modern SI. It is loosely inspired by the brain: many simple units, or “neurons”, are connected in layers, and each connection has a strength, called a weight.

How it works

1. Input enters the first layer, such as the pixels of an image or tokens of text.
2. Each neuron combines its inputs, weighted by connection strengths, and passes a signal to the next layer.
3. The output layer produces a prediction, such as “cat” or the next word.
4. During training, the network compares its prediction to the right answer and nudges millions or billions of weights to reduce the error. The algorithm that works out these nudges is called backpropagation.

The adjustable weights are the model’s parameters. Frontier models have hundreds of billions to trillions of them.

A short history

• 1943: Warren McCulloch and Walter Pitts describe a mathematical model of a neuron.
• 1958: Frank Rosenblatt builds the perceptron, an early learning machine.
• 1986: Rumelhart, Hinton and Williams popularise backpropagation.
• 2012 onward: deep learning makes very large networks practical.

Types you’ll hear about

Convolutional networks (images), recurrent networks (older language and speech models) and transformers (today’s language and multimodal models).