Weight quantization is the process of representing a neural network's learned parameters at a lower numerical precision than the full 32-bit or 16-bit floating-point format used during training. In the most common scheme — post-training quantization — weights are mapped to 8-bit integers (INT8) or even 4-bit integers (INT4) after training is complete, with a small calibration step to determine the scaling factors that minimise the rounding error introduced. The practical effect is a 2–8× reduction in model size and memory bandwidth, which translates directly into faster inference and the ability to run larger models on consumer hardware.

Think of it like converting a precise engineering blueprint that uses measurements to six decimal places into a version that rounds to the nearest millimetre. For most construction purposes the rounded version builds the same structure at a fraction of the storage cost, with only negligible loss in fidelity for the application at hand.

Quantizing a model with the popular bitsandbytes library requires only a flag change at load time:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    load_in_4bit=True,          # INT4 quantization
    device_map="auto",
)

This loads a model that would otherwise require ~16 GB of GPU memory into roughly 5 GB, making it runnable on a single consumer GPU.

Weight quantization is closely related to Quantization as a general concept, Inference optimisation, and Knowledge Distillation — another compression technique that achieves similar deployment benefits through a different mechanism. Parameter count determines the baseline memory footprint that quantization then reduces, and LoRA adapters can be trained in low-precision regimes using a technique called QLoRA.

You will encounter weight quantization whenever you deploy open-source language models locally, run inference on edge devices, or need to fit a larger model into a constrained GPU budget in production.