Neural networks: compose learned transformations

A neural network applies a sequence of parameterized operations to an input. Training changes the parameters so the final output becomes useful for a chosen objective. A layer is a computational building block, not a miniature human neuron.

One common layer multiplies a vector by a weight matrix, adds a bias and applies an activation. Each output combines information from several input components. Later layers transform those combinations again.

Why the nonlinearity matters

If every layer were only an affine transformation, the whole stack could be collapsed into one affine transformation. More layers would change the parameterization but not the class of functions it could express. Nonlinear activations break that restriction.

This NumPy network has one hidden layer. x contains input rows; w1 and w2 map input to hidden width and hidden width to output. The biases broadcast across rows. The function accepts supplied weights rather than training them.

import numpy as np

def network(x, w1, b1, w2, b2):
    hidden = np.maximum(x @ w1 + b1, 0)
    return hidden @ w2 + b2

Even a small network can represent a relationship that one linear boundary cannot. For example, two suitable hidden units can distinguish whether exactly one of two binary inputs is active.

The same principle, different layers

Modern language models combine learned transformations with attention, normalization and residual connections. Convolutional networks reuse local filters; recurrent networks carry state across steps. The arrangement determines which computations are easy to represent and run.

Backpropagation connects the final loss to the parameters that produced it. Architecture supplies the possible computations; training selects their numerical behaviour.