AI Model Wiki

    Skip to main content
    20 min
    Lesson 5 of 7

    Performance & Cost Optimization

    From the course AI Fine-Tuning Pipelines

    Reduce training time and inference costs without sacrificing model quality.

    Training Optimizations

    Flash Attention 2

    Flash Attention 2 rewrites the attention kernel to be IO-aware, giving 2-4x speedup:

    model = AutoModelForCausalLM.from_pretrained(
        "meta-llama/Llama-2-7b-hf",
        attn_implementation="flash_attention_2",
        torch_dtype=torch.bfloat16,
        device_map="auto"
    )
    

    Gradient Checkpointing

    Trade compute for memory — essential for fine-tuning on consumer GPUs:

    model.gradient_checkpointing_enable()
    # Reduces VRAM by ~40% at ~20% slower training speed
    

    Mixed Precision Training

    training_args = TrainingArguments(
        bf16=True,          # Ampere+ GPUs (A100, 3090)
        # fp16=True,        # Older GPUs (V100, 2080 Ti)
        tf32=True,          # A100 matrix math acceleration
    )
    

    Inference Optimizations

    vLLM for High-Throughput Serving

    from vllm import LLM, SamplingParams
    
    llm = LLM(model="./merged-support-model", tensor_parallel_size=1)
    params = SamplingParams(temperature=0.7, max_tokens=256)
    
    # Process 100 requests in a single batch
    prompts = [build_prompt(q) for q in question_batch]
    outputs = llm.generate(prompts, params)
    # vLLM achieves 10-30x higher throughput vs HuggingFace generate()
    

    Quantization for Deployment

    MethodSize ReductionQuality LossLatency
    FP1650%NoneBaseline
    INT875%Minimal-20%
    INT4 (GPTQ)87.5%Small-35%
    INT4 (AWQ)87.5%Very small-30%

    AWQ Quantization (Best Quality/Speed)

    from awq import AutoAWQForCausalLM
    
    model = AutoAWQForCausalLM.from_pretrained("./fine-tuned-model")
    quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}
    model.quantize(tokenizer, quant_config=quant_config)
    model.save_quantized("./fine-tuned-model-awq")
    # Result: 7B model goes from 14GB → 4GB with minimal quality drop
    

    Cost Benchmarks

    Training 7B model on 10K examples:

    • A100 80GB (cloud): ~$8, ~2 hours
    • RTX 4090 (owned): ~$0.50 electricity, ~3 hours
    • OpenAI fine-tuning: ~$30 (gpt-3.5-turbo)

    Inference at 100 requests/minute:

    • vLLM on A10G: ~$0.0002/request
    • OpenAI gpt-3.5-turbo: ~$0.002/request (10x more expensive)

    Key Takeaways

    • •Flash Attention 2 provides 2-4x training speedup with no code changes beyond one flag
    • •AWQ quantization shrinks a 7B model from 14GB to 4GB with minimal quality loss
    • •vLLM achieves 10-30x higher throughput than HuggingFace generate() for batch inference
    • •Self-hosted inference at $0.0002/request vs OpenAI at $0.002/request — 10x cheaper at scale
    0 of 7 lessons marked complete

    Progress is kept in this browser only.