16 min read
Lesson 4 of 5
Training, Monitoring & Evaluation
From the course Fine-Tuning AI Models: LoRA, QLoRA & Beyond
Training a fine-tuned model is iterative — you'll rarely get it right on the first try. Knowing what to monitor and how to evaluate is critical.
Training Hyperparameters
- Learning rate: Start with 2e-4 for LoRA. Too high causes instability; too low wastes time.
- Batch size: Larger is more stable but uses more memory. Use gradient accumulation to simulate larger batches.
- Epochs: 1-3 epochs for most tasks. More risks overfitting.
- Warmup steps: Gradually increase learning rate for the first 5-10% of training to avoid early instability.
- Weight decay: Light regularization (0.01) helps prevent overfitting.
Monitoring Training
Use Weights & Biases (W&B) or TensorBoard to track:
- Training loss: Should decrease steadily. Spikes indicate learning rate issues.
- Validation loss: Should decrease alongside training loss. Divergence signals overfitting.
- Learning rate schedule: Verify your warmup and decay are working as expected.
- GPU utilization: Should be near 100%. Low utilization means you can increase batch size.
Evaluation Strategies
Automated metrics only tell part of the story:
- Perplexity: Lower is better. Measures how surprised the model is by correct answers.
- BLEU/ROUGE: For text generation tasks. Compare generated text to reference outputs.
- Task-specific accuracy: Classification accuracy, extraction F1, etc.
- Human evaluation: The gold standard. Have domain experts rate outputs on relevance, accuracy, and quality.
A/B Testing
The ultimate test is real-world performance:
- Deploy both the base model (with prompting) and your fine-tuned model
- Route a percentage of traffic to each
- Measure user satisfaction, task completion rate, and error rate
- Only fully switch when the fine-tuned model demonstrably wins
Common Training Issues
| Symptom | Likely Cause | Fix |
|---|---|---|
| Loss doesn't decrease | Learning rate too low | Increase 2-5× |
| Loss spikes randomly | Learning rate too high | Decrease by half |
| Val loss increases while train loss drops | Overfitting | Reduce epochs, add dropout |
| Outputs are repetitive | Temperature too low or overfitting | Adjust inference params, reduce training |
| Model forgets base capabilities | Catastrophic forgetting | Lower learning rate, fewer epochs |
Key Takeaways
- •Start with learning rate 2e-4, 1-3 epochs, and monitor validation loss for overfitting
- •Human evaluation remains the gold standard — automated metrics are necessary but insufficient
- •A/B testing against prompted base models validates real-world improvement
- •Catastrophic forgetting is prevented by low learning rates and limited training duration
0 of 5 lessons marked complete
Progress is kept in this browser only.