FLOPS — Floating Point Operations Per Second — is the standard unit for measuring raw computational throughput in AI hardware and training runs. One petaFLOP equals one quadrillion floating-point operations per second. When researchers describe training a model at a certain scale, they typically express the total compute budget in petaFLOP-days (petaFLOPS sustained for 24 hours) or as a single exponent of total FLOP count, allowing apples-to-apples comparison across different hardware and durations.

Think of it like measuring how fast a construction crew can lay bricks. FLOPS tells you the theoretical maximum rate — but just as crew size doesn't account for coordination, breaks, or bad weather, peak FLOPS rarely reflects what hardware actually delivers on a real training job, where memory bandwidth, interconnects, and software efficiency all impose practical limits (called utilization or MFU, model FLOPS utilization).

A concrete example: training GPT-3 required roughly 3.14 × 10²³ total FLOPS. An NVIDIA H100 SXM5 GPU delivers about 1,979 teraFLOPS (fp16 dense), so in theory a cluster of 1,000 H100s running at 50% utilization would finish that workload in around three days. The training compute for frontier models like GPT-4 is estimated at roughly 2 × 10²⁵ FLOPS, more than 60× larger.

FLOPS is closely linked to concepts like Inference (which also consumes FLOPS per forward pass), Quantization (which trades precision for FLOPS savings), and Parameter count (larger models require more FLOPS per token). Knowledge Distillation is partly motivated by the desire to reduce inference FLOPS without sacrificing quality.

You will encounter FLOPS whenever comparing GPU or TPU hardware specifications, reading AI scaling law papers, or estimating cloud compute costs for a training or fine-tuning run.