Speculative decoding is an Inference acceleration technique that pairs a small, fast "draft" model with a large, accurate "target" model to generate text faster than the target model can alone. The draft model autoregressively proposes a short sequence of tokens in rapid succession; the target model then evaluates all proposed tokens in a single parallel forward pass, accepting correct ones and rejecting the first incorrect one. Because the target model's expensive forward pass processes multiple tokens at once instead of one at a time, effective throughput increases substantially—often two to four times—with no change to output quality.

Think of it like a knowledgeable research assistant who drafts a paragraph while their senior colleague is busy. The senior colleague glances at the draft, stamps each correct sentence with approval, and rewrites only from the first mistake onward. The finished page is entirely the senior colleague's work, but the process was far quicker than dictating every word from scratch.

A typical implementation uses a smaller model of the same family—say a 7B draft alongside a 70B target. In Hugging Face Transformers this is exposed as: outputs = target_model.generate(inputs, assistant_model=draft_model, do_sample=False). The library handles proposal, verification, and rollback transparently.

Speculative decoding is most relevant when discussing LLM serving latency, Quantization trade-offs, and Knowledge Distillation—the draft model is often a distilled version of the target. You will encounter it in production inference stacks such as vLLM and TGI, and in benchmarks comparing tokens-per-second across hardware configurations.