19 min
Lesson 3 of 4
MLflow & Production ML
From the course Databricks AI & ML
MLflow is the backbone of production ML on Databricks. Master it to manage the full lifecycle of your AI models.
Experiment Tracking
import mlflow
with mlflow.start_run():
mlflow.log_param("model", "llama-3-70b")
mlflow.log_param("temperature", 0.7)
mlflow.log_param("chunk_size", 500)
# Run evaluation
results = evaluate_rag_pipeline(chain, test_questions)
mlflow.log_metric("accuracy", results["accuracy"])
mlflow.log_metric("latency_p50", results["latency_p50"])
mlflow.log_metric("relevance_score", results["relevance"])
mlflow.langchain.log_model(chain, "rag_chain")
Model Evaluation
MLflow provides AI-specific evaluation:
results = mlflow.evaluate(
model=chain,
data=eval_dataset,
model_type="question-answering",
evaluators=["default"],
extra_metrics=[
mlflow.metrics.genai.relevance(),
mlflow.metrics.genai.faithfulness(),
mlflow.metrics.toxicity()
]
)
Metrics include: relevance, faithfulness, toxicity, answer similarity.
Model Registry
Lifecycle Management
- Development: Initial model logged from experiments
- Staging: Promoted for testing and validation
- Production: Approved for serving live traffic
- Archived: Retired models kept for compliance
Deployment Patterns
- A/B Testing: Route traffic between model versions
- Canary: Gradually shift traffic to new models
- Shadow: Run new models alongside production without serving
Monitoring
Lakehouse Monitoring
- Track prediction drift over time
- Monitor feature distributions
- Alert on model quality degradation
- Log all inference requests for audit
Cost Tracking
- Monitor GPU compute usage per model
- Track token consumption for LLM endpoints
- Set budgets and alerts per team
- Optimize serving instance sizes based on traffic
Best Practices
- Use Unity Catalog for model governance
- Tag all experiments with team, project, and purpose
- Automate model retraining pipelines
- Implement circuit breakers for external AI dependencies
- Document model decisions and limitations
Key Takeaways
- •MLflow tracks experiments, metrics, and model artifacts for reproducibility
- •AI-specific evaluation includes relevance, faithfulness, and toxicity metrics
- •Model Registry manages the lifecycle from development through production to archive
- •Lakehouse Monitoring tracks prediction drift and alerts on quality degradation
0 of 4 lessons marked complete
Progress is kept in this browser only.