AI Model Wiki

    Skip to main content
    45 min
    Lesson 1 of 5

    Testing AI Applications

    From the course AI Testing & Quality Assurance

    Why AI Testing Is Different

    AI applications are non-deterministic — the same input can produce different outputs. Traditional unit testing (exact match) doesn't work. Instead, we test for properties, patterns, and quality ranges.

    Testing Pyramid for AI

             /  E2E Tests  \        (Full pipeline)
            /  Integration   \      (API + model)
           /  Component Tests  \    (Prompts, parsing)
          /  Unit Tests          \  (Functions, schemas)
    

    Types of AI Tests

    • Deterministic tests: Validate schemas, parsing, tool call formats
    • Property tests: Check outputs have required properties (length, format, tone)
    • Comparison tests: Verify output quality vs. a baseline
    • Regression tests: Ensure changes don't degrade known-good outputs

    Deterministic Testing

    // Test that structured output matches schema
    test("product extraction returns valid JSON", async () => {
      const result = await extractProduct("iPhone 15 Pro, $999, 256GB");
      expect(result).toHaveProperty("name");
      expect(result).toHaveProperty("price");
      expect(typeof result.price).toBe("number");
      expect(result.price).toBeGreaterThan(0);
    });
    

    Property-Based Testing

    test("summary is shorter than input", async () => {
      const input = generateLongText(5000);
      const summary = await summarize(input);
      expect(summary.length).toBeLessThan(input.length * 0.3);
      expect(summary.length).toBeGreaterThan(50);
    });
    

    Common Pitfall: Treating Failed Tests as Passing When Retrying

    AI outputs are non-deterministic — a test may fail once and pass on retry. Running a failing test three times and accepting the best result masks real quality issues. For CI, accept a failure rate threshold instead:

    async function runWithThreshold(testFn: () => Promise<boolean>, runs = 5, threshold = 0.8) {
      const results = await Promise.all(Array.from({ length: runs }, testFn));
      const passRate = results.filter(Boolean).length / runs;
      return passRate >= threshold;
    }
    
    const passed = await runWithThreshold(() => testSentimentClassification(), 5, 0.8);
    // Expects 4 out of 5 runs to pass — a stricter measure than "passes sometimes"
    

    Best Practice: Use a Dedicated Test Model That's Cheaper

    When running test suites in CI, use a cheaper model (gpt-4o-mini or claude-haiku) for all deterministic and property tests. Reserve expensive models (gpt-4o, claude-sonnet) only for quality evaluation tests that specifically test the production model. This can reduce CI costs by 80%:

    const TEST_MODEL = process.env.CI ? "gpt-4o-mini" : "gpt-4o";
    const EVAL_MODEL = "gpt-4o";  // Always use the production model for quality evals
    

    Named Product Reference: Vitest for AI Unit Tests

    Vitest is a fast TypeScript-native test runner (from the Vite ecosystem) that supports mocking ES modules natively — important for mocking AI provider SDKs. Its vi.mock function handles dynamic imports and allows you to swap the real OpenAI client for a deterministic mock without configuration complexity. For Next.js AI projects, Vitest is generally preferable to Jest due to better ESM support.

    Key Takeaways

    • •AI outputs are non-deterministic — test properties, not exact values
    • •Use a testing pyramid: unit → component → integration → E2E
    • •Deterministic tests validate schemas, formats, and tool calls
    • •Property tests check output characteristics (length, format, tone)
    0 of 5 lessons marked complete

    Progress is kept in this browser only.