Quantization is a model compression technique that represents a neural network's weights and activations using lower-precision numerical formats - typically reducing from 32-bit floating point (FP32) down to 16-bit, 8-bit, or even 4-bit integers - thereby shrinking memory footprint and accelerating Inference at the cost of a small, often acceptable drop in accuracy. The core trade-off is precision against efficiency: fewer bits per value means faster matrix multiplications and less memory bandwidth, but the coarser numerical grid introduces rounding errors that can accumulate across the many layers of a large Neural Network. Post-training quantization (PTQ) applies to an already-trained model without retraining; quantization-aware training (QAT) simulates low-precision arithmetic during the training loop so the model learns to compensate.

Think of it like converting a high-resolution photograph to a lower-resolution version for a thumbnail: most viewers cannot tell the difference at a glance, the file is a fraction of the size, and it loads much faster - but if you zoom in, the compression artefacts become visible.

Loading a model in 8-bit with Hugging Face BitsAndBytes:

from transformers import AutoModelForCausalLM

# load in 8-bit -- roughly halves VRAM vs FP16
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    load_in_8bit=True,
    device_map="auto"
)
# inference unchanged; tokenize and call model.generate() as normal

Quantization pairs naturally with Pruning and Knowledge Distillation in compression pipelines, and is the enabler behind QLoRA, which loads a 4-bit quantised base for fine-tuning. It also interacts with LoRA adapters, which are kept in full precision atop a quantised backbone. Understanding Parameter count is prerequisite to appreciating why reducing bits per parameter matters at scale.

You will encounter quantization any time you deploy a model on a resource-constrained device - a laptop GPU, an edge chip, or a cost-sensitive cloud API - and it is the primary reason that 70-billion-parameter LLMs can run on a single high-end consumer GPU today.