20 min
Lesson 5 of 7
Performance & Cost Optimization
From the course AI Fine-Tuning Pipelines
Reduce training time and inference costs without sacrificing model quality.
Training Optimizations
Flash Attention 2
Flash Attention 2 rewrites the attention kernel to be IO-aware, giving 2-4x speedup:
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
attn_implementation="flash_attention_2",
torch_dtype=torch.bfloat16,
device_map="auto"
)
Gradient Checkpointing
Trade compute for memory — essential for fine-tuning on consumer GPUs:
model.gradient_checkpointing_enable()
# Reduces VRAM by ~40% at ~20% slower training speed
Mixed Precision Training
training_args = TrainingArguments(
bf16=True, # Ampere+ GPUs (A100, 3090)
# fp16=True, # Older GPUs (V100, 2080 Ti)
tf32=True, # A100 matrix math acceleration
)
Inference Optimizations
vLLM for High-Throughput Serving
from vllm import LLM, SamplingParams
llm = LLM(model="./merged-support-model", tensor_parallel_size=1)
params = SamplingParams(temperature=0.7, max_tokens=256)
# Process 100 requests in a single batch
prompts = [build_prompt(q) for q in question_batch]
outputs = llm.generate(prompts, params)
# vLLM achieves 10-30x higher throughput vs HuggingFace generate()
Quantization for Deployment
| Method | Size Reduction | Quality Loss | Latency |
|---|---|---|---|
| FP16 | 50% | None | Baseline |
| INT8 | 75% | Minimal | -20% |
| INT4 (GPTQ) | 87.5% | Small | -35% |
| INT4 (AWQ) | 87.5% | Very small | -30% |
AWQ Quantization (Best Quality/Speed)
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_pretrained("./fine-tuned-model")
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized("./fine-tuned-model-awq")
# Result: 7B model goes from 14GB → 4GB with minimal quality drop
Cost Benchmarks
Training 7B model on 10K examples:
- A100 80GB (cloud): ~$8, ~2 hours
- RTX 4090 (owned): ~$0.50 electricity, ~3 hours
- OpenAI fine-tuning: ~$30 (gpt-3.5-turbo)
Inference at 100 requests/minute:
- vLLM on A10G: ~$0.0002/request
- OpenAI gpt-3.5-turbo: ~$0.002/request (10x more expensive)
Key Takeaways
- •Flash Attention 2 provides 2-4x training speedup with no code changes beyond one flag
- •AWQ quantization shrinks a 7B model from 14GB to 4GB with minimal quality loss
- •vLLM achieves 10-30x higher throughput than HuggingFace generate() for batch inference
- •Self-hosted inference at $0.0002/request vs OpenAI at $0.002/request — 10x cheaper at scale
0 of 7 lessons marked complete
Progress is kept in this browser only.