CI/CD for ML extends traditional software CI/CD with model testing, data validation, and quality gates specific to AI systems.

The ML CI/CD Pipeline

Code Change → Lint/Test → Data Validation → Model Training → Evaluation → Staging → Production

Each stage has ML-specific concerns:

Code testing (same as traditional):

• Unit tests for data preprocessing, feature engineering, postprocessing
• Integration tests for pipeline components
• Linting and type checking

Data validation:

• Schema validation (expected columns, types, ranges)
• Distribution checks (no major shifts from baseline)
• Quality checks (completeness, duplicates, outliers)
• Great Expectations, Pandera, or TFX Data Validation

Model evaluation gates:

• Run evaluation suite on new model
• Compare against current production model
• Require improvement on key metrics (or at least no regression)
• Human review for significant changes

Staged rollout:

• Deploy to staging environment first
• Run shadow mode (process production traffic but don't serve results)
• Gradual traffic shifting (1% → 10% → 50% → 100%)
• Automatic rollback if quality metrics degrade

Infrastructure as Code

Define your serving infrastructure declaratively:

# model-deployment.yaml
model:
  name: customer-support-v3
  framework: vllm
  gpu: a10g
  replicas:
    min: 1
    max: 5
  scaling:
    metric: queue_depth
    target: 10
  health_check:
    endpoint: /health
    interval: 30s
  rollback:
    on_error_rate: 5%
    window: 15m

Model Versioning

Track every model version with:

• Training dataset version and hash
• Hyperparameters and configuration
• Evaluation metrics on standardized test sets
• Git commit hash of training code
• Training infrastructure and duration
• Deployment history and performance in production

Use MLflow, Weights & Biases, or a simple metadata store to maintain this lineage.

Testing AI Differently

Traditional tests assert exact outputs. AI tests assert properties:

def test_summarization_length():
    """Summary should be shorter than input."""
    summary = model.summarize(long_document)
    assert len(summary) < len(long_document) * 0.3

def test_classification_confidence():
    """High-confidence predictions should be accurate."""
    for example in high_confidence_examples:
        result = model.classify(example.input)
        assert result.label == example.expected_label

def test_safety_refusal():
    """Model should refuse harmful requests."""
    for prompt in harmful_prompts:
        response = model.generate(prompt)
        assert not contains_harmful_content(response)