Testing AI Applications
From the course AI Testing & Quality Assurance
Why AI Testing Is Different
AI applications are non-deterministic — the same input can produce different outputs. Traditional unit testing (exact match) doesn't work. Instead, we test for properties, patterns, and quality ranges.
Testing Pyramid for AI
/ E2E Tests \ (Full pipeline)
/ Integration \ (API + model)
/ Component Tests \ (Prompts, parsing)
/ Unit Tests \ (Functions, schemas)
Types of AI Tests
- Deterministic tests: Validate schemas, parsing, tool call formats
- Property tests: Check outputs have required properties (length, format, tone)
- Comparison tests: Verify output quality vs. a baseline
- Regression tests: Ensure changes don't degrade known-good outputs
Deterministic Testing
// Test that structured output matches schema
test("product extraction returns valid JSON", async () => {
const result = await extractProduct("iPhone 15 Pro, $999, 256GB");
expect(result).toHaveProperty("name");
expect(result).toHaveProperty("price");
expect(typeof result.price).toBe("number");
expect(result.price).toBeGreaterThan(0);
});
Property-Based Testing
test("summary is shorter than input", async () => {
const input = generateLongText(5000);
const summary = await summarize(input);
expect(summary.length).toBeLessThan(input.length * 0.3);
expect(summary.length).toBeGreaterThan(50);
});
Common Pitfall: Treating Failed Tests as Passing When Retrying
AI outputs are non-deterministic — a test may fail once and pass on retry. Running a failing test three times and accepting the best result masks real quality issues. For CI, accept a failure rate threshold instead:
async function runWithThreshold(testFn: () => Promise<boolean>, runs = 5, threshold = 0.8) {
const results = await Promise.all(Array.from({ length: runs }, testFn));
const passRate = results.filter(Boolean).length / runs;
return passRate >= threshold;
}
const passed = await runWithThreshold(() => testSentimentClassification(), 5, 0.8);
// Expects 4 out of 5 runs to pass — a stricter measure than "passes sometimes"
Best Practice: Use a Dedicated Test Model That's Cheaper
When running test suites in CI, use a cheaper model (gpt-4o-mini or claude-haiku) for all deterministic and property tests. Reserve expensive models (gpt-4o, claude-sonnet) only for quality evaluation tests that specifically test the production model. This can reduce CI costs by 80%:
const TEST_MODEL = process.env.CI ? "gpt-4o-mini" : "gpt-4o";
const EVAL_MODEL = "gpt-4o"; // Always use the production model for quality evals
Named Product Reference: Vitest for AI Unit Tests
Vitest is a fast TypeScript-native test runner (from the Vite ecosystem) that supports mocking ES modules natively — important for mocking AI provider SDKs. Its vi.mock function handles dynamic imports and allows you to swap the real OpenAI client for a deterministic mock without configuration complexity. For Next.js AI projects, Vitest is generally preferable to Jest due to better ESM support.
Key Takeaways
- •AI outputs are non-deterministic — test properties, not exact values
- •Use a testing pyramid: unit → component → integration → E2E
- •Deterministic tests validate schemas, formats, and tool calls
- •Property tests check output characteristics (length, format, tone)
Progress is kept in this browser only.