Compute Optimal Training refers to the practice of allocating a fixed computational budget—measured in floating-point operations (FLOPs)—across model size and training data volume so as to minimize the final Loss Function value. The concept was formalized by the Chinchilla scaling laws (DeepMind, 2022), which showed that the optimal allocation scales model parameters and training tokens at roughly a 1:20 ratio. Deviating in either direction—training a very large model on too little data, or a small model on enormous data—yields a worse result for the same compute cost.

Think of it like allocating study hours between reading new material and practicing problems: too much time reading without practice, or too much practice on too few concepts, both leave you less prepared than a balanced split would.

The Chinchilla paper demonstrated this by training a suite of models across a sweep of size and data combinations, then fitting a parametric loss curve. The 70B Chinchilla model, trained on 1.4 trillion tokens, outperformed the 280B Gopher model trained on 300 billion tokens—despite using similar total FLOPs. Meta's LLaMA models applied this insight, training 7B–65B models on 1–1.4 trillion tokens, making them far more efficient to serve at Inference time.

Compute Optimal Training connects to Chinchilla Scaling, Hyperparameter selection, Pretraining strategy, and the downstream economics of Knowledge Distillation and Quantization for deployment. It also informs Foundation Model design, since a compute-optimal base model requires fewer Parameters to reach a given capability level, reducing memory and latency costs for every downstream use. Fine-tuning decisions often depend on whether the base model was compute-optimal or undertrained.

Practitioners encounter Compute Optimal Training when planning large pretraining runs, selecting open-weight models for fine-tuning, and evaluating the efficiency claims in model release papers.