A fine-tuned model sitting on your laptop isn't useful. Getting it into production requires careful consideration of serving infrastructure, merging strategies, and monitoring.

Merging Adapters

LoRA adapters can be merged back into the base model for simpler deployment:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
model = PeftModel.from_pretrained(base_model, "path/to/adapter")
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged-model")

Alternatively, keep adapters separate to:

• Swap adapters for different tasks on the same base model
• A/B test different fine-tunes without loading multiple full models
• Roll back to the base model instantly

Serving Options

1. vLLM: High-throughput serving with PagedAttention. Best for production workloads.
2. Text Generation Inference (TGI): Hugging Face's production server. Easy setup with Docker.
3. Ollama: For local/edge deployment. Simple but less optimized.
4. Cloud APIs: Upload to Together AI, Fireworks, or Modal for managed serving.

Quantization for Deployment

Reduce model size for faster inference:

• GGUF format: For llama.cpp and Ollama. Various quantization levels (Q4_K_M is a good default).
• AWQ: Activation-aware quantization. Better quality than naive 4-bit.
• GPTQ: Post-training quantization with calibration data.

Production Monitoring

Once deployed, continuously track:

• Latency: Time to first token and total generation time
• Throughput: Requests per second at various batch sizes
• Quality drift: Periodic human evaluation to catch degradation
• Cost per query: GPU hours × cloud pricing
• Error rates: Failed generations, timeouts, safety filter triggers

Version Management

Treat fine-tuned models like software releases:

• Tag each training run with dataset version, hyperparameters, and metrics
• Store adapters in a model registry (Hugging Face Hub, MLflow)
• Maintain rollback capability to previous versions
• Document what changed between versions and why