AI Model Wiki

    Skip to main content
    22 min
    Lesson 2 of 4

    Automated AI Testing

    From the course AI CI/CD Pipelines

    Implement automated testing strategies for AI components in your CI pipeline.

    Test Pyramid for AI

                        ▲
                       /│\  E2E Tests (few, expensive)
                      / │ \   - Full user flows with AI
                     /  │  \  - Production traffic replay
                    /───┼───\
                   /    │    \  Integration Tests
                  /     │     \   - API contract tests
                 /      │      \  - Model serving tests
                /───────┼───────\
               /        │        \  Unit Tests (many, fast)
              /         │         \   - Prompt template tests
             /          │          \  - Output parser tests
            ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
    

    GitHub Actions AI Pipeline

    name: AI CI Pipeline
    
    on: [push, pull_request]
    
    jobs:
      unit-tests:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - name: Run unit tests
            run: pytest tests/unit -v
            
      prompt-tests:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - name: Test prompt templates
            run: python -m pytest tests/prompts --tb=short
            env:
              OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
              
      evaluation:
        needs: [unit-tests, prompt-tests]
        runs-on: ubuntu-latest
        steps:
          - name: Run evaluation suite
            run: |
              python evaluate.py \
                --dataset tests/eval_dataset.json \
                --output results.json
                
          - name: Check quality gates
            run: |
              python check_gates.py results.json \
                --min-accuracy 0.85 \
                --max-latency-p99 3000
    

    Prompt Regression Tests

    import pytest
    from ai_app import generate_response
    
    @pytest.mark.parametrize("test_case", load_test_cases("prompt_tests.yaml"))
    def test_prompt_behavior(test_case):
        response = generate_response(test_case["input"])
        
        # Check required content
        for required in test_case.get("must_contain", []):
            assert required.lower() in response.lower()
        
        # Check forbidden content
        for forbidden in test_case.get("must_not_contain", []):
            assert forbidden.lower() not in response.lower()
        
        # Check semantic similarity
        if "reference_response" in test_case:
            similarity = compute_similarity(response, test_case["reference_response"])
            assert similarity >= test_case.get("min_similarity", 0.8)
    

    Key Takeaways

    • •AI test pyramid: many fast unit tests, fewer integration tests, minimal E2E tests
    • •GitHub Actions can run prompt tests and evaluation suites with quality gates
    • •Prompt regression tests check required/forbidden content and semantic similarity
    • •Quality gates enforce minimum accuracy and maximum latency before deployment
    0 of 4 lessons marked complete

    Progress is kept in this browser only.