AI Model Wiki

    Skip to main content
    40 min
    Lesson 2 of 5

    Cold Start Optimization

    From the course Serverless AI Deployment

    Minimizing Cold Start Latency

    What Causes Cold Starts?

    1. Container initialization (OS, runtime)
    2. Code loading (dependencies, modules)
    3. Connection establishment (database, API clients)
    4. Model loading (weights, tokenizers)

    Optimization Strategies

    Reduce Package Size:

    • Bundle only used modules (tree-shaking)
    • Use lighter alternatives (e.g., openai SDK vs full langchain)
    • Exclude test files, docs, and type definitions from deployment

    Warm Pools:

    • Use provisioned concurrency (AWS Lambda) for guaranteed warm instances
    • Schedule periodic pings to keep functions warm
    • Accept first-request latency for low-traffic functions

    Lazy Loading:

    let openaiClient: OpenAI | null = null;
    
    function getClient() {
      if (!openaiClient) {
        openaiClient = new OpenAI({ apiKey: Deno.env.get("OPENAI_API_KEY") });
      }
      return openaiClient;
    }
    

    Connection Reuse:

    • Initialize database connections outside the handler
    • Reuse HTTP clients across invocations
    • Cache DNS lookups and TLS sessions

    Platform-Specific Tips

    • Supabase Edge Functions: Deno runtime, near-zero cold starts
    • AWS Lambda: Use arm64 architecture for better price-performance
    • Cloudflare Workers: V8 isolates have near-instant cold starts
    • Vercel: Edge runtime for fastest starts, Node.js for heavier compute

    Common Pitfall: Importing Heavy Dependencies at Module Level

    Lambda cold starts are dominated by module load time. Importing "@anthropic-ai/sdk", "langchain", or "@huggingface/inference" at the top of the file means they are parsed and initialized on every cold start — even if the invocation doesn't use them. Use lazy imports instead:

    // BAD — imported at module load time
    import Anthropic from "@anthropic-ai/sdk";
    
    // GOOD — imported only when needed
    export async function handler(event: any) {
      const { default: Anthropic } = await import("@anthropic-ai/sdk");
      const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
      // ...
    }
    

    Best Practice: Pre-Warm Connections Outside the Handler

    Lambda reuses the execution environment across invocations (within the same container). Any code executed at module level — database pool creation, SDK instantiation, config loading — persists between invocations. Initializing your OpenAI client outside the handler body means subsequent requests within the same container skip that cost entirely:

    // Runs once per container, reused across invocations
    import OpenAI from "openai";
    const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! });
    
    export async function handler() {
      // 'client' is already initialized
      const result = await client.chat.completions.create({ ... });
    }
    

    Named Product Reference: AWS Lambda Provisioned Concurrency

    AWS Lambda Provisioned Concurrency keeps a specified number of function instances initialized and warm at all times — eliminating cold starts for those instances. For AI API-proxy functions that must respond in under 200ms (e.g., a chatbot backend), provisioned concurrency is the production-grade solution. AWS charges per GB-second of provisioned capacity regardless of invocations, so it is best paired with auto-scaling policies that track request volume and provision accordingly.

    Key Takeaways

    • •Cold starts come from container init, code loading, and connections
    • •Minimize package size and use lazy loading for faster starts
    • •Reuse connections and clients across function invocations
    • •Edge runtimes (Deno, V8) have near-zero cold starts
    0 of 5 lessons marked complete

    Progress is kept in this browser only.