A parameter is a numerical value stored inside a neural network that is adjusted during training to minimize the model's loss on its training data. Parameters are the actual learned content of the model: in a transformer-based language model, they encode the associations, syntax patterns, factual knowledge, and reasoning heuristics extracted from the training corpus. They are distinct from hyperparameters, which are configuration choices set before training begins and not learned from data. The total parameter count of a model — often reported in billions — is one of the primary indicators of its capacity, though not the only one.

Think of it like the settings on a complex mixing board in a recording studio. Each dial and slider starts at a default position, and an audio engineer (the training algorithm) turns thousands of knobs simultaneously, listening to the result and nudging each one until the output sounds right. The final positions of all those dials are the parameters — the distilled knowledge of how to mix that kind of music.

In a feed-forward layer, parameters are the weight matrices and bias vectors that transform input vectors into output vectors:

import torch.nn as nn
layer = nn.Linear(768, 3072)   # weight: 768×3072 = 2,359,296 params; bias: 3,072 params
total = sum(p.numel() for p in layer.parameters())
print(f"{total:,} parameters")  # 2,362,368

A GPT-style model with 7 billion parameters contains roughly that many floating-point numbers, each consuming 2 bytes in FP16 — about 14 GB of storage before any runtime overhead. This is why Quantization (reducing numeric precision) and LoRA (updating only a small fraction of parameters during fine-tuning) are so practically important.

Parameters are the fundamental unit shaped by Gradient Descent through a Loss Function, and their collective behavior determines what the Neural Network has learned. Overfitting often results when a model has far more parameters than training examples can meaningfully constrain.

You will encounter parameter counts whenever comparing model sizes, estimating GPU memory requirements, or choosing between a large capable model and a smaller, faster one for a latency-sensitive application.