Deep learning: learn the intermediate representations too

Deep learning uses neural networks with multiple learned processing layers. Each layer transforms the previous representation, so training can shape both the final predictor and the features it relies on.

For images, successive operations can combine local visual patterns into larger structures. In a Transformer, attention mixes information between positions while feed-forward layers transform the representation at each position.

Depth needs more than repeated multiplication

Two linear maps in succession can be collapsed into one linear map. Inserting nonlinear operations lets the stack represent relationships that a single linear transformation cannot.

Depth creates a compositional way to build complex functions, but adding layers alone does not guarantee better results. Training signal, architecture, data and optimization determine whether the extra capacity is useful.

The important distinction from manually engineered features is where the representation comes from. The network learns intermediate transformations through the task's objective rather than receiving every useful feature from a programmer.