AI Model Wiki

    Skip to main content
    22 min
    Lesson 3 of 7

    Evaluation & Deployment

    From the course AI Fine-Tuning Pipelines

    Evaluate fine-tuned models and deploy them to production.

    Model Evaluation

    Automated Evaluation

    from lm_eval import evaluator
    
    results = evaluator.simple_evaluate(
        model="hf",
        model_args="pretrained=./my-finetuned-model",
        tasks=["hellaswag", "arc_challenge", "mmlu"],
        num_fewshot=5
    )
    

    Task-Specific Evaluation

    def evaluate_fine_tuned_model(model, test_set):
        metrics = {
            "exact_match": 0,
            "semantic_similarity": 0,
            "format_compliance": 0,
            "factual_accuracy": 0,
        }
        
        for example in test_set:
            output = model.generate(example["input"])
            metrics["exact_match"] += exact_match(output, example["expected"])
            metrics["semantic_similarity"] += semantic_sim(output, example["expected"])
        
        return {k: v / len(test_set) for k, v in metrics.items()}
    

    Deployment Options

    Merge LoRA Weights

    from peft import PeftModel
    
    base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
    peft_model = PeftModel.from_pretrained(base_model, "./lora-weights")
    
    merged_model = peft_model.merge_and_unload()
    merged_model.save_pretrained("./merged-model")
    

    A/B Testing Fine-Tuned Models

    async def route_request(request):
        if random.random() < 0.1:  # 10% to fine-tuned
            response = await fine_tuned_model(request)
            log_experiment("fine_tuned", request, response)
        else:
            response = await base_model(request)
            log_experiment("base", request, response)
        return response
    

    Key Takeaways

    • •LM Eval Harness provides standardized benchmarks (HellaSwag, ARC, MMLU)
    • •Task-specific evaluation measures exact match, semantic similarity, and factual accuracy
    • •Merge LoRA weights into base model for simplified deployment
    • •vLLM with tensor parallelism enables efficient production serving of fine-tuned models
    0 of 7 lessons marked complete

    Progress is kept in this browser only.