AI Model Wiki

    Skip to main content
    40 min
    Lesson 3 of 5

    Regression Testing & CI/CD

    From the course AI Testing & Quality Assurance

    Preventing Quality Regressions

    Snapshot Testing for AI

    // Save baseline outputs
    const baseline = await runEvalSuite(currentPrompt);
    saveBaseline("v1.0", baseline);
    
    // After changes, compare
    const updated = await runEvalSuite(newPrompt);
    const regression = compareResults(baseline, updated);
    
    if (regression.degradedCases > regression.improvedCases) {
      throw new Error(`Regression detected: ${regression.degradedCases} cases worse`);
    }
    

    CI/CD Integration

    # GitHub Actions workflow
    name: AI Quality Gate
    on: [pull_request]
    
    jobs:
      eval:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - name: Run AI evaluations
            run: npm run eval
            env:
              OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          - name: Check quality threshold
            run: |
              SCORE=$(cat eval-results.json | jq '.averageScore')
              if (( $(echo "$SCORE < 0.8" | bc -l) )); then
                echo "Quality below threshold: $SCORE"
                exit 1
              fi
    

    Cost-Effective CI Testing

    • Use cheaper models (GPT-4o-mini) for CI evals
    • Cache API responses for deterministic tests
    • Run full evals nightly, quick checks on PRs
    • Use mock responses for integration tests

    Version Control for Prompts

    • Store prompts in version-controlled files
    • Track prompt changes with git history
    • Link prompt versions to eval results
    • Roll back to previous prompt versions if quality drops

    Common Pitfall: Running Expensive Evals on Every Commit

    Full evaluation suites using GPT-4o as a judge can cost $5–$50 per run. Running these on every commit creates an unsustainable CI cost. Split into tiers:

    # Fast tier: runs on every commit (~$0.10)
    - name: Fast evals
      run: npm run eval:fast  # 20 deterministic tests, cheap model
    
    # Quality tier: runs on PRs to main (~$2)
    - name: Quality evals
      if: github.base_ref == 'main'
      run: npm run eval:quality  # 100 tests, GPT-4o judge
    
    # Full tier: runs nightly (~$20)
    - name: Full evals
      if: github.event_name == 'schedule'
      run: npm run eval:full  # 500 tests, full suite
    

    Best Practice: Create Eval Fixtures for Regression Testing

    When a user reports a bad AI output, add it to your eval dataset immediately. This converts bug reports into permanent regression tests:

    // After receiving a bug report:
    await evalDataset.addCase({
      input: "Reported bad input from user ticket #4521",
      expectedProperties: ["mentions pricing", "does not mention competitors"],
      source: "bug_report",
      ticketId: "4521",
      addedAt: new Date().toISOString(),
    });
    

    Named Product Reference: Promptfoo for Prompt Testing

    Promptfoo is an open-source CLI tool for testing and comparing prompts. Define test cases in YAML, run them against multiple models simultaneously, and get a visual diff of results. It supports LLM-as-judge evaluations, factuality checks, and custom assertion functions. For teams iterating on prompts, promptfoo eval provides a faster feedback loop than running a full test suite.

    Key Takeaways

    • •Snapshot testing compares new outputs against a known-good baseline
    • •Integrate AI evaluations into CI/CD with quality gates
    • •Use cheaper models and caching for cost-effective CI testing
    • •Version-control prompts and link them to evaluation results
    0 of 5 lessons marked complete

    Progress is kept in this browser only.