Cold Start Optimization
From the course Serverless AI Deployment
Minimizing Cold Start Latency
What Causes Cold Starts?
- Container initialization (OS, runtime)
- Code loading (dependencies, modules)
- Connection establishment (database, API clients)
- Model loading (weights, tokenizers)
Optimization Strategies
Reduce Package Size:
- Bundle only used modules (tree-shaking)
- Use lighter alternatives (e.g.,
openaiSDK vs fulllangchain) - Exclude test files, docs, and type definitions from deployment
Warm Pools:
- Use provisioned concurrency (AWS Lambda) for guaranteed warm instances
- Schedule periodic pings to keep functions warm
- Accept first-request latency for low-traffic functions
Lazy Loading:
let openaiClient: OpenAI | null = null;
function getClient() {
if (!openaiClient) {
openaiClient = new OpenAI({ apiKey: Deno.env.get("OPENAI_API_KEY") });
}
return openaiClient;
}
Connection Reuse:
- Initialize database connections outside the handler
- Reuse HTTP clients across invocations
- Cache DNS lookups and TLS sessions
Platform-Specific Tips
- Supabase Edge Functions: Deno runtime, near-zero cold starts
- AWS Lambda: Use arm64 architecture for better price-performance
- Cloudflare Workers: V8 isolates have near-instant cold starts
- Vercel: Edge runtime for fastest starts, Node.js for heavier compute
Common Pitfall: Importing Heavy Dependencies at Module Level
Lambda cold starts are dominated by module load time. Importing "@anthropic-ai/sdk", "langchain", or "@huggingface/inference" at the top of the file means they are parsed and initialized on every cold start — even if the invocation doesn't use them. Use lazy imports instead:
// BAD — imported at module load time
import Anthropic from "@anthropic-ai/sdk";
// GOOD — imported only when needed
export async function handler(event: any) {
const { default: Anthropic } = await import("@anthropic-ai/sdk");
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
// ...
}
Best Practice: Pre-Warm Connections Outside the Handler
Lambda reuses the execution environment across invocations (within the same container). Any code executed at module level — database pool creation, SDK instantiation, config loading — persists between invocations. Initializing your OpenAI client outside the handler body means subsequent requests within the same container skip that cost entirely:
// Runs once per container, reused across invocations
import OpenAI from "openai";
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! });
export async function handler() {
// 'client' is already initialized
const result = await client.chat.completions.create({ ... });
}
Named Product Reference: AWS Lambda Provisioned Concurrency
AWS Lambda Provisioned Concurrency keeps a specified number of function instances initialized and warm at all times — eliminating cold starts for those instances. For AI API-proxy functions that must respond in under 200ms (e.g., a chatbot backend), provisioned concurrency is the production-grade solution. AWS charges per GB-second of provisioned capacity regardless of invocations, so it is best paired with auto-scaling policies that track request volume and provision accordingly.
Key Takeaways
- •Cold starts come from container init, code loading, and connections
- •Minimize package size and use lazy loading for faster starts
- •Reuse connections and clients across function invocations
- •Edge runtimes (Deno, V8) have near-zero cold starts
Progress is kept in this browser only.