AI Model Wiki

    Skip to main content
    40 min
    Lesson 3 of 5

    Cost Management & Scaling

    From the course Serverless AI Deployment

    Controlling Serverless AI Costs

    Cost Components

    1. Compute: Function execution time × memory
    2. API calls: AI provider charges per token
    3. Data transfer: Egress charges for responses
    4. Storage: Caching, queues, databases

    Cost Optimization

    Right-size memory:

    • More memory = faster execution but higher cost per ms
    • Profile your function to find the optimal memory/speed tradeoff
    • AI proxy functions typically need 256-512MB

    Minimize execution time:

    • Use streaming to free the function as soon as the stream starts
    • Cache frequent responses (Redis, CDN)
    • Set timeouts to prevent runaway executions

    Rate limiting:

    // Simple in-memory rate limiter
    const requests = new Map<string, number[]>();
    
    function isRateLimited(userId: string, limit = 10, windowMs = 60000): boolean {
      const now = Date.now();
      const userRequests = requests.get(userId)?.filter(t => now - t < windowMs) || [];
      if (userRequests.length >= limit) return true;
      userRequests.push(now);
      requests.set(userId, userRequests);
      return false;
    }
    

    Common Pitfall: Not Separating Compute Costs from AI API Costs

    Serverless billing dashboards show you Lambda/Edge function costs, but the AI provider charges (OpenAI tokens, Anthropic tokens) appear in a completely separate account and can be 10–100x larger than the compute costs. Teams routinely hit surprise $500–$2000 AI API bills because they only monitored Lambda spend. Tag all AI requests with a shared tracing ID and track token consumption in your own database alongside compute metrics.

    Best Practice: Right-Size Memory Using Lambda Power Tuning

    AWS Lambda's execution time and cost are both affected by memory allocation. Adding more memory speeds up the function (more CPU is allocated proportionally), and there is often a "sweet spot" where more memory reduces total cost because the function finishes much faster. Use the open-source AWS Lambda Power Tuning tool (a Step Functions state machine) to automatically benchmark your function at multiple memory settings and plot the cost-performance curve.

    Named Product Reference: Upstash Redis for Serverless Rate Limiting

    Upstash (upstash.com) provides serverless Redis with per-request pricing — no idle cost — making it ideal for serverless AI rate limiting. Their "@upstash/ratelimit" library offers sliding window, fixed window, and token bucket algorithms with a single SDK call. Unlike in-memory rate limiting (which resets per Lambda container), Upstash Redis is shared across all containers:

    import { Ratelimit } from "@upstash/ratelimit";
    import { Redis } from "@upstash/redis";
    
    const ratelimit = new Ratelimit({
      redis: Redis.fromEnv(),
      limiter: Ratelimit.slidingWindow(10, "60 s"),
    });
    
    const { success } = await ratelimit.limit(userId);
    if (!success) return new Response("Too Many Requests", { status: 429 });
    

    Monitoring

    • Track invocation count, duration, and errors
    • Set billing alerts before hitting unexpected costs
    • Monitor AI API usage separately from compute costs
    • Use dashboards to visualize cost trends over time

    Key Takeaways

    • •Serverless costs come from compute, API calls, transfer, and storage
    • •Right-size function memory for optimal cost-performance balance
    • •Implement rate limiting to prevent abuse and cost overruns
    • •Monitor and alert on both compute and AI API costs
    0 of 5 lessons marked complete

    Progress is kept in this browser only.