Production Patterns & Reliability
From the course Serverless AI Deployment
Building Reliable Serverless AI
Retry Patterns
async function callWithRetry(fn: () => Promise<any>, maxRetries = 3) {
for (let i = 0; i < maxRetries; i++) {
try {
return await fn();
} catch (error) {
if (i === maxRetries - 1) throw error;
const delay = Math.min(1000 * Math.pow(2, i), 10000);
await new Promise(r => setTimeout(r, delay + Math.random() * 1000));
}
}
}
Circuit Breaker
Prevent cascading failures when AI providers are down:
class CircuitBreaker {
private failures = 0;
private lastFailure = 0;
private readonly threshold = 5;
private readonly resetTime = 60000;
async call<T>(fn: () => Promise<T>): Promise<T> {
if (this.failures >= this.threshold && Date.now() - this.lastFailure < this.resetTime) {
throw new Error("Circuit breaker open — service unavailable");
}
try {
const result = await fn();
this.failures = 0;
return result;
} catch (error) {
this.failures++;
this.lastFailure = Date.now();
throw error;
}
}
}
Multi-Provider Fallback
async function aiComplete(prompt: string) {
try {
return await callOpenAI(prompt);
} catch {
try {
return await callAnthropic(prompt);
} catch {
return await callGemini(prompt);
}
}
}
Observability
- Structured logging with request IDs
- Distributed tracing across function chains
- Error tracking with automatic alerting
- Performance dashboards with P50/P95/P99 latency
Common Pitfall: Retrying Non-Idempotent AI Operations
Retrying an AI generation call that already completed but whose response was lost in transit can cause the same action to be executed twice (sending an email, creating a record, posting a message). Before adding retry logic to any AI-triggered operation, classify it: read operations and pure generation are safe to retry; state-mutation operations need idempotency keys or deduplication at the application layer.
Best Practice: Multi-Provider Fallback with Timeout Budgets
A cascading fallback chain (OpenAI → Anthropic → Gemini) is only useful if each fallback has an independent timeout. Otherwise a hanging OpenAI request consumes most of your Lambda timeout before Anthropic is even tried. Set a per-provider timeout of 20–30 seconds and let the fallback chain fire immediately on either error or timeout:
async function aiCompleteWithBudget(prompt: string) {
const providers = [callOpenAI, callAnthropic, callGemini];
for (const provider of providers) {
try {
return await Promise.race([
provider(prompt),
new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), 25000)),
]);
} catch {
continue;
}
}
throw new Error("All providers failed");
}
Named Product Reference: Axiom for Serverless AI Observability
Axiom (axiom.co) provides structured log ingestion and querying optimized for serverless workloads. Unlike CloudWatch Logs (which charges for storage and queries), Axiom's free tier supports unlimited events up to 90-day retention. Their "@axiomhq/js" SDK works in both Deno/Edge and Node.js Lambda environments. For AI applications, you can log every request's token count, model, latency, and user ID in a single structured event and then build dashboards showing cost-per-user and P95 latency by model.
Key Takeaways
- •Circuit breakers prevent cascading failures during outages
- •Multi-provider fallback ensures availability
- •Structured logging and tracing are essential for debugging
Progress is kept in this browser only.