Dropout: make the network work with missing activations

Dropout randomly removes activations during training. The network cannot rely on exactly the same set being present at every step, which can reduce overfitting.

In the common inverted-dropout convention, surviving activations are divided by the keep probability. Their expectation stays unchanged; an individual sample's sum does not have to stay unchanged.

A mask changes each training pass

This NumPy implementation exposes the operation. The same input receives different masks on two passes.

import numpy as np

rng = np.random.default_rng(7)
def dropout(hidden, rate):
    keep = rng.random(hidden.shape) >= rate
    return hidden * keep / (1 - rate)

hidden = np.ones((4, 8))
first = dropout(hidden, 0.25)
second = dropout(hidden, 0.25)

At ordinary inference, dropout is disabled and the activation passes through unchanged. Framework training/evaluation mode therefore matters even when the weights are identical.

Dropout does not reduce the number of stored parameters and is not the same as permanently pruning a network. Its rate is a training choice: too much can prevent useful features from being learned. The helpful rate depends on the model, data and other regularization.