Reduce model size and speed up inference through quantization.

Quantization Overview

Precision Levels

Post-Training Quantization (PTQ)

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Dynamic quantization (easiest)
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},
    dtype=torch.qint8
)

# Static quantization (better quality)
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
torch.quantization.prepare(model, inplace=True)
# Run calibration data through model
torch.quantization.convert(model, inplace=True)

GPTQ Quantization

from auto_gptq import AutoGPTQForCausalLM

# Load pre-quantized model
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-7B-GPTQ",
    device="cuda:0",
    use_safetensors=True
)

# 4-bit inference with minimal quality loss
output = model.generate(input_ids)

AWQ (Activation-Aware Quantization)

Benefits over GPTQ:

• Faster inference
• Better quality preservation
• Activation-aware scaling