AI Model Wiki

    Skip to main content
    19 min
    Lesson 3 of 4

    MLflow & Production ML

    From the course Databricks AI & ML

    MLflow is the backbone of production ML on Databricks. Master it to manage the full lifecycle of your AI models.

    Experiment Tracking

    import mlflow
    
    with mlflow.start_run():
        mlflow.log_param("model", "llama-3-70b")
        mlflow.log_param("temperature", 0.7)
        mlflow.log_param("chunk_size", 500)
        
        # Run evaluation
        results = evaluate_rag_pipeline(chain, test_questions)
        
        mlflow.log_metric("accuracy", results["accuracy"])
        mlflow.log_metric("latency_p50", results["latency_p50"])
        mlflow.log_metric("relevance_score", results["relevance"])
        
        mlflow.langchain.log_model(chain, "rag_chain")
    

    Model Evaluation

    MLflow provides AI-specific evaluation:

    results = mlflow.evaluate(
        model=chain,
        data=eval_dataset,
        model_type="question-answering",
        evaluators=["default"],
        extra_metrics=[
            mlflow.metrics.genai.relevance(),
            mlflow.metrics.genai.faithfulness(),
            mlflow.metrics.toxicity()
        ]
    )
    

    Metrics include: relevance, faithfulness, toxicity, answer similarity.

    Model Registry

    Lifecycle Management

    1. Development: Initial model logged from experiments
    2. Staging: Promoted for testing and validation
    3. Production: Approved for serving live traffic
    4. Archived: Retired models kept for compliance

    Deployment Patterns

    • A/B Testing: Route traffic between model versions
    • Canary: Gradually shift traffic to new models
    • Shadow: Run new models alongside production without serving

    Monitoring

    Lakehouse Monitoring

    • Track prediction drift over time
    • Monitor feature distributions
    • Alert on model quality degradation
    • Log all inference requests for audit

    Cost Tracking

    • Monitor GPU compute usage per model
    • Track token consumption for LLM endpoints
    • Set budgets and alerts per team
    • Optimize serving instance sizes based on traffic

    Best Practices

    • Use Unity Catalog for model governance
    • Tag all experiments with team, project, and purpose
    • Automate model retraining pipelines
    • Implement circuit breakers for external AI dependencies
    • Document model decisions and limitations

    Key Takeaways

    • •MLflow tracks experiments, metrics, and model artifacts for reproducibility
    • •AI-specific evaluation includes relevance, faithfulness, and toxicity metrics
    • •Model Registry manages the lifecycle from development through production to archive
    • •Lakehouse Monitoring tracks prediction drift and alerts on quality degradation
    0 of 4 lessons marked complete

    Progress is kept in this browser only.