Scaling laws describe the empirical relationship between a language model's performance and the three main resources poured into training it: compute (measured in FLOPs), dataset size (tokens), and parameter count. Research from OpenAI and later DeepMind's Chinchilla team showed these relationships follow clean power laws — doubling compute roughly halves loss by a predictable margin — and that the three factors must be scaled together in specific ratios to get the best return on investment. The Chinchilla result in particular overturned earlier assumptions, demonstrating that many large models were undertrained relative to their size.

Think of it like the physics of baking bread: there are reliable ratios between flour, water, yeast, and time. Ignore one dimension and the loaf doesn't scale with the others no matter how much more you add of everything else.

# Chinchilla-optimal compute allocation (rough rule of thumb)
# For a model of N parameters, train on ~20N tokens
N = 7e9          # 7 billion parameters
tokens = 20 * N  # approximately 140 billion tokens
# Llama-2 7B was trained on 2T tokens — deliberately over-trained
# for cheaper inference, an intentional deviation from the scaling law

Scaling laws underpin decisions about Pretraining budgets and Foundation Model design; they interact with Quantization and Knowledge Distillation strategies that trade off model size against inference cost. Mixture of Experts architectures are partly motivated by reaching better loss at lower active-parameter count, a direct application of scaling law thinking.

You will encounter scaling laws in almost every paper announcing a new large model — they set the expectations that practitioners use to decide how much to spend and what performance to anticipate before training begins.