22 min
Lesson 3 of 7
Evaluation & Deployment
From the course AI Fine-Tuning Pipelines
Evaluate fine-tuned models and deploy them to production.
Model Evaluation
Automated Evaluation
from lm_eval import evaluator
results = evaluator.simple_evaluate(
model="hf",
model_args="pretrained=./my-finetuned-model",
tasks=["hellaswag", "arc_challenge", "mmlu"],
num_fewshot=5
)
Task-Specific Evaluation
def evaluate_fine_tuned_model(model, test_set):
metrics = {
"exact_match": 0,
"semantic_similarity": 0,
"format_compliance": 0,
"factual_accuracy": 0,
}
for example in test_set:
output = model.generate(example["input"])
metrics["exact_match"] += exact_match(output, example["expected"])
metrics["semantic_similarity"] += semantic_sim(output, example["expected"])
return {k: v / len(test_set) for k, v in metrics.items()}
Deployment Options
Merge LoRA Weights
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
peft_model = PeftModel.from_pretrained(base_model, "./lora-weights")
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("./merged-model")
A/B Testing Fine-Tuned Models
async def route_request(request):
if random.random() < 0.1: # 10% to fine-tuned
response = await fine_tuned_model(request)
log_experiment("fine_tuned", request, response)
else:
response = await base_model(request)
log_experiment("base", request, response)
return response
Key Takeaways
- •LM Eval Harness provides standardized benchmarks (HellaSwag, ARC, MMLU)
- •Task-specific evaluation measures exact match, semantic similarity, and factual accuracy
- •Merge LoRA weights into base model for simplified deployment
- •vLLM with tensor parallelism enables efficient production serving of fine-tuned models
0 of 7 lessons marked complete
Progress is kept in this browser only.