Regression Testing & CI/CD
From the course AI Testing & Quality Assurance
Preventing Quality Regressions
Snapshot Testing for AI
// Save baseline outputs
const baseline = await runEvalSuite(currentPrompt);
saveBaseline("v1.0", baseline);
// After changes, compare
const updated = await runEvalSuite(newPrompt);
const regression = compareResults(baseline, updated);
if (regression.degradedCases > regression.improvedCases) {
throw new Error(`Regression detected: ${regression.degradedCases} cases worse`);
}
CI/CD Integration
# GitHub Actions workflow
name: AI Quality Gate
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run AI evaluations
run: npm run eval
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: Check quality threshold
run: |
SCORE=$(cat eval-results.json | jq '.averageScore')
if (( $(echo "$SCORE < 0.8" | bc -l) )); then
echo "Quality below threshold: $SCORE"
exit 1
fi
Cost-Effective CI Testing
- Use cheaper models (GPT-4o-mini) for CI evals
- Cache API responses for deterministic tests
- Run full evals nightly, quick checks on PRs
- Use mock responses for integration tests
Version Control for Prompts
- Store prompts in version-controlled files
- Track prompt changes with git history
- Link prompt versions to eval results
- Roll back to previous prompt versions if quality drops
Common Pitfall: Running Expensive Evals on Every Commit
Full evaluation suites using GPT-4o as a judge can cost $5–$50 per run. Running these on every commit creates an unsustainable CI cost. Split into tiers:
# Fast tier: runs on every commit (~$0.10)
- name: Fast evals
run: npm run eval:fast # 20 deterministic tests, cheap model
# Quality tier: runs on PRs to main (~$2)
- name: Quality evals
if: github.base_ref == 'main'
run: npm run eval:quality # 100 tests, GPT-4o judge
# Full tier: runs nightly (~$20)
- name: Full evals
if: github.event_name == 'schedule'
run: npm run eval:full # 500 tests, full suite
Best Practice: Create Eval Fixtures for Regression Testing
When a user reports a bad AI output, add it to your eval dataset immediately. This converts bug reports into permanent regression tests:
// After receiving a bug report:
await evalDataset.addCase({
input: "Reported bad input from user ticket #4521",
expectedProperties: ["mentions pricing", "does not mention competitors"],
source: "bug_report",
ticketId: "4521",
addedAt: new Date().toISOString(),
});
Named Product Reference: Promptfoo for Prompt Testing
Promptfoo is an open-source CLI tool for testing and comparing prompts. Define test cases in YAML, run them against multiple models simultaneously, and get a visual diff of results. It supports LLM-as-judge evaluations, factuality checks, and custom assertion functions. For teams iterating on prompts, promptfoo eval provides a faster feedback loop than running a full test suite.
Key Takeaways
- •Snapshot testing compares new outputs against a known-good baseline
- •Integrate AI evaluations into CI/CD with quality gates
- •Use cheaper models and caching for cost-effective CI testing
- •Version-control prompts and link them to evaluation results
Progress is kept in this browser only.