Inference is the phase of a machine learning model's lifecycle in which the trained model receives new input and produces an output — a prediction, a generated text sequence, a classification label, or an image. It is distinct from training, where the model's weights are updated through gradient descent; during inference, weights are frozen and the model simply executes a forward pass through its layers. For large neural networks, inference involves billions of floating-point multiplications per response, making hardware selection, batching strategy, and precision format (such as Quantization to INT8 or INT4) critical engineering decisions.

Think of it like the difference between studying for an exam and actually sitting it: training is the months of study where knowledge is internalized, and inference is the moment you answer questions using what you've already learned — no new studying happens during the exam.

In a production LLM deployment, inference typically works like this: a user sends a prompt via API, the text is converted to tokens through Tokenization, the token IDs are passed through the Transformer's attention layers, and the model outputs a probability distribution over the vocabulary at each step, from which the next token is sampled. On a single A100 GPU, a 70-billion-parameter model running in 16-bit precision might generate 20-30 tokens per second; quantizing to 4-bit roughly doubles throughput at some quality cost.

Inference cost and latency are directly affected by Context Window length (longer contexts require more memory per request), by the use of speculative decoding and caching techniques, and by model architecture choices like Mixture of Experts (which activates only a subset of parameters per token). Inference is contrasted with Pretraining and Fine-tuning in discussions of the full model lifecycle.

You will encounter inference as the central topic in any conversation about deploying AI in production — including cloud API costs, on-device model running, streaming response latency, and GPU memory budgeting.