Neural scaling laws are empirically derived relationships - typically following power laws - that describe how a model's performance improves as one or more of three quantities increases: the number of trainable Parameters, the size of the training dataset measured in tokens, and the total compute budget measured in floating-point operations. The landmark Chinchilla paper (Hoffmann et al., 2022) refined earlier OpenAI scaling work by showing that compute-optimal training requires model size and token count to scale roughly in proportion, upending the then-common practice of training very large models on relatively small datasets.

Think of it like the relationship between hours of practice and skill level in a sport: more practice reliably improves performance, the gains per extra hour follow a diminishing curve that is highly predictable, and knowing the curve lets coaches plan training schedules in advance without running every experiment.

A simplified power-law fit for test loss L as a function of model parameters N looks like:

import numpy as np

# Empirical coefficients from Kaplan et al. (2020) for language modelling
A, alpha = 406.4, 0.034   # constants for parameter scaling

def predicted_loss(N):
    return A * (N ** -alpha)

print(predicted_loss(1e9))   # ~2.82 nats for a 1B-param model
print(predicted_loss(1e11))  # ~2.37 nats for a 100B-param model

Because the relationships are smooth and predictable across many orders of magnitude, practitioners use scaling laws to forecast the performance of a model before training it, allocating compute budgets rationally. Related concepts include Pretraining (the stage at which scaling laws are measured), Perplexity (the metric commonly plotted against scale), LLM development (where trillion-parameter ambitions are directly guided by these laws), and Inference cost (which scaling laws do not directly optimise, motivating techniques like Quantization and Knowledge Distillation).

You will encounter neural scaling laws whenever you read about decisions behind frontier model training runs, compute-optimal model selection, or the economic case for investing in larger datasets versus larger architectures.