Fine-tuning adapts a pre-trained model to your specific use case. LoRA and QLoRA make this accessible on consumer hardware.

What is Fine-Tuning?

Fine-tuning takes a pre-trained model and trains it further on your specific data. This teaches the model:

• Your domain's terminology and concepts
• Your preferred writing style and tone
• Task-specific patterns (classification, extraction, formatting)
• Company-specific knowledge and conventions

Full Fine-Tuning vs LoRA

Full fine-tuning updates every parameter in the model:

• Requires enormous GPU resources (often multiple A100s)
• Creates a complete new model copy
• Can cause "catastrophic forgetting" (model loses general abilities)
• Impractical for most individuals and small teams

LoRA (Low-Rank Adaptation) adds small trainable matrices to the model:

• Only trains 0.1-1% of the total parameters
• Requires a single consumer GPU (8-24GB VRAM)
• Creates a small adapter file (10-100MB) instead of a full model copy
• Can be swapped in and out without modifying the base model
• Preserves the model's general capabilities

QLoRA combines LoRA with quantization:

• Loads the base model in 4-bit precision
• Trains LoRA adapters in 16-bit precision
• Fine-tune a 7B model with just 6GB VRAM
• Fine-tune a 13B model with 12GB VRAM
• Quality is nearly identical to full LoRA

When to Fine-Tune (and When Not To)

✅ Fine-tune when:

• You need consistent output formatting
• RAG alone doesn't capture the nuance of your domain
• You want the model to "think" like a domain expert
• You have 100+ high-quality training examples
• You need to reduce prompt length (bake instructions into the model)

❌ Don't fine-tune when:

• RAG can solve your problem (try RAG first!)
• You need factual knowledge updates (use RAG instead)
• You have fewer than 50 training examples
• You need the model to handle diverse, general tasks

Preparing Training Data

Fine-tuning data format (conversation style):

[
  {
    "messages": [
      {"role": "system", "content": "You are a medical coding assistant."},
      {"role": "user", "content": "Patient presents with acute bronchitis."},
      {"role": "assistant", "content": "ICD-10: J20.9 - Acute bronchitis, unspecified"}
    ]
  },
  // ... 100-1000 more examples
]

Data quality rules:

• Minimum 50 examples, ideally 200-1000
• Diverse examples covering edge cases
• Consistent formatting across all examples
• Human-verified outputs (never use AI-generated training data without review)
• Balance across categories/classes

Fine-Tuning Tools

For Beginners:

• Unsloth — 2x faster training, 60% less memory. Best tool for consumer hardware.
• Axolotl — Simple YAML configuration, supports many model architectures
• Hugging Face AutoTrain — No-code fine-tuning in the browser

For Teams:

• OpenAI Fine-Tuning API — Upload data, get a fine-tuned model (cloud-based)
• Together AI — Cloud fine-tuning with open-source models
• Modal / Lambda — Rent GPUs by the hour for training runs

A Typical Fine-Tuning Workflow:

1. Collect and clean training data (100-1000 examples)
2. Split into train (90%) and validation (10%) sets
3. Choose a base model (Llama 3, Mistral, Phi-3)
4. Configure LoRA parameters (rank, alpha, target modules)
5. Train for 1-3 epochs (monitor validation loss)
6. Test the fine-tuned model qualitatively
7. Merge LoRA adapter with base model (optional)
8. Deploy locally or convert to GGUF for Ollama