Activation functions: make the transformation nonlinear

An activation function changes a layer's numerical response. Without nonlinear operations, a chain of affine layers remains one affine transformation. Activations let a network represent richer relationships while shaping how gradients travel during training.

ReLU keeps positive inputs and maps negative inputs to zero. GELU and SiLU, common Transformer ingredients, apply smoother responses. SiLU is also called Swish when using its usual fixed scale.

A gate that does not simply switch off

SiLU multiplies an input by its sigmoid. The sigmoid factor lies between zero and one, but the final SiLU output can be negative. It is a smooth weighting of the input, not a hard on/off decision.

import numpy as np

def silu(x):
    return x / (1 + np.exp(-x))

For a moderately negative input, ReLU gives zero while SiLU retains a small negative value. Their derivatives differ too. Those local differences affect which parameter updates a network can receive.

Activations can control another branch

A gated feed-forward network computes two projections. One passes through an activation and multiplies the other element by element. The result is then projected back to the model width. SwiGLU uses SiLU in that gate.

This is different from softmax: softmax couples candidates through a shared normalization, whereas the elementwise activation here treats components individually.

An activation does not learn a task by itself. Its role depends on the surrounding weights, the training objective and the architecture. The interesting choice is how its response and gradient interact with the computation around it.