An activation function is a mathematical operation applied to a neuron's output that determines whether and how strongly that neuron "fires," introducing non-linearity into the network. Without this non-linearity, stacking multiple layers would still produce only a linear transformation, severely limiting what a model can learn. Common choices include ReLU (Rectified Linear Unit), GELU, SiLU, and Softmax, each with different properties suited to different parts of a network.

Think of it like a light dimmer rather than a simple on/off switch. A linear system can only brighten or dim proportionally, but a non-linear activation lets the network say "ignore anything below this threshold" or "compress extreme values"—giving it the flexibility to model curved, complex relationships in data.

In a transformer-based language model, the feed-forward sub-layer typically applies GELU between two linear projections:

import torch.nn.functional as F
# x is the hidden state after the first linear layer
x = F.gelu(linear1(x))
output = linear2(x)

The choice of activation affects training stability, gradient flow, and final accuracy. ReLU is fast and sparse; GELU is smoother and favored in modern LLMs.

Activation functions are closely tied to Gradient Descent (they must be differentiable for backpropagation), Loss Function design, Neural Network architecture, and Transformer construction. Understanding them is also prerequisite to reading about Attention mechanisms, which rely on Softmax to normalize scores.

You will encounter activation function choices in model architecture papers, framework documentation, and fine-tuning guides—particularly when authors justify switching from ReLU to GELU or SwiGLU in a new model version.