22 min
Lesson 2 of 4
Automated AI Testing
From the course AI CI/CD Pipelines
Implement automated testing strategies for AI components in your CI pipeline.
Test Pyramid for AI
▲
/│\ E2E Tests (few, expensive)
/ │ \ - Full user flows with AI
/ │ \ - Production traffic replay
/───┼───\
/ │ \ Integration Tests
/ │ \ - API contract tests
/ │ \ - Model serving tests
/───────┼───────\
/ │ \ Unit Tests (many, fast)
/ │ \ - Prompt template tests
/ │ \ - Output parser tests
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
GitHub Actions AI Pipeline
name: AI CI Pipeline
on: [push, pull_request]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run unit tests
run: pytest tests/unit -v
prompt-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Test prompt templates
run: python -m pytest tests/prompts --tb=short
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
evaluation:
needs: [unit-tests, prompt-tests]
runs-on: ubuntu-latest
steps:
- name: Run evaluation suite
run: |
python evaluate.py \
--dataset tests/eval_dataset.json \
--output results.json
- name: Check quality gates
run: |
python check_gates.py results.json \
--min-accuracy 0.85 \
--max-latency-p99 3000
Prompt Regression Tests
import pytest
from ai_app import generate_response
@pytest.mark.parametrize("test_case", load_test_cases("prompt_tests.yaml"))
def test_prompt_behavior(test_case):
response = generate_response(test_case["input"])
# Check required content
for required in test_case.get("must_contain", []):
assert required.lower() in response.lower()
# Check forbidden content
for forbidden in test_case.get("must_not_contain", []):
assert forbidden.lower() not in response.lower()
# Check semantic similarity
if "reference_response" in test_case:
similarity = compute_similarity(response, test_case["reference_response"])
assert similarity >= test_case.get("min_similarity", 0.8)
Key Takeaways
- •AI test pyramid: many fast unit tests, fewer integration tests, minimal E2E tests
- •GitHub Actions can run prompt tests and evaluation suites with quality gates
- •Prompt regression tests check required/forbidden content and semantic similarity
- •Quality gates enforce minimum accuracy and maximum latency before deployment
0 of 4 lessons marked complete
Progress is kept in this browser only.