LoRA (Low-Rank Adaptation) revolutionized fine-tuning by making it accessible on consumer hardware. QLoRA pushed this further with quantization.

How LoRA Works

Instead of updating all model weights, LoRA freezes the pre-trained model and injects small trainable matrices (adapters) into each transformer layer. These adapters capture task-specific knowledge while the base model retains its general capabilities.

Mathematically, instead of learning a full weight update ΔW (millions of parameters), LoRA decomposes it into two small matrices: ΔW = A × B, where A and B have far fewer parameters.

Key LoRA Hyperparameters

• Rank (r): The dimensionality of the adapter matrices. Higher rank = more capacity but more memory. Common values: 8, 16, 32, 64.
• Alpha (α): Scaling factor for the adapter. Usually set to 2× the rank. Controls how much the adapters influence the output.
• Target modules: Which layers to add adapters to. Typically attention layers (q_proj, v_proj, k_proj, o_proj) and sometimes MLP layers.
• Dropout: Regularization to prevent overfitting. Usually 0.05-0.1.

QLoRA: Quantized LoRA

QLoRA combines LoRA with 4-bit quantization of the base model:

1. The base model is loaded in 4-bit precision (NF4 quantization)
2. LoRA adapters are trained in 16-bit precision
3. Gradients flow through the quantized base model via double quantization

This means you can fine-tune a 70B parameter model on a single 24GB GPU — previously impossible.

Practical Setup with Hugging Face

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# QLoRA config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    quantization_config=bnb_config,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

Choosing Rank

• r=8: Good starting point for simple tasks (classification, formatting)
• r=16-32: Sweet spot for most fine-tuning tasks
• r=64+: Only needed for complex domain adaptation

Higher rank doesn't always mean better results — it can lead to overfitting on small datasets.