AI Model Wiki

    Skip to main content
    40 min
    Lesson 3 of 5

    TTL Management & Monitoring

    From the course Prompt Caching & Cost Optimization

    Managing Cache Lifetime

    TTL Strategies

    Anthropic (5-minute refreshing):

    • Cache refreshes on each hit — stays alive as long as you use it
    • For bursty traffic, send periodic requests to keep cache warm
    • For low-traffic, accept cache misses and rebuild

    Google Gemini (custom TTL):

    • Set explicit TTL from minutes to hours
    • Longer TTL = higher storage costs but fewer cache misses
    • Delete caches when no longer needed

    OpenAI (automatic):

    • No TTL management needed — fully automatic
    • Caching happens transparently for prompts >1,024 tokens
    • Check usage API for cache hit statistics

    Monitoring Cache Performance

    Track these metrics:

    • Cache hit rate: Percentage of requests that hit the cache
    • Cost savings: Compare actual spend vs. uncached estimate
    • Latency improvement: Time to first token with vs. without cache
    • Cache waste: Tokens cached but rarely reused

    Cost Modeling

    Build a simple model to evaluate caching ROI:

    monthly_savings = (cached_tokens × cache_read_discount × request_count)
                      - (cached_tokens × cache_write_premium × cache_creates)
                      - (storage_cost × hours_cached)
    

    If monthly_savings > 0, caching is worth it.

    Common Pitfall: Not Accounting for Cache Write Cost

    Caching is not free — providers charge a "cache write" premium (Anthropic charges 25% more than the normal input token price to create a cache entry). If you create a new cache entry on every request (because your cached prefix keeps changing), you pay the write premium on every call without getting the read discount. Monitor the ratio of cache writes to reads:

    const metrics = {
      cache_writes: response.usage.cache_creation_input_tokens,
      cache_reads: response.usage.cache_read_input_tokens,
    };
    
    const ratio = metrics.cache_reads / (metrics.cache_writes + metrics.cache_reads);
    console.log(`Cache efficiency: ${(ratio * 100).toFixed(1)}%`);
    // Target: >80% reads, <20% writes for positive ROI
    

    Best Practice: Combine Prompt Caching with Response Caching

    For frequently asked questions, add a second caching layer that stores the final LLM response. If a user asks "What is your return policy?" for the tenth time today, return the cached answer directly — no LLM call at all. Hash the user's message (after normalization) as the cache key:

    const cacheKey = `response:${hashQuery(userMessage)}`;
    const cached = await redis.get(cacheKey);
    if (cached) return JSON.parse(cached);
    
    const response = await callLLM(systemPrompt, userMessage);
    await redis.setex(cacheKey, 3600, JSON.stringify(response));
    return response;
    

    Named Product Reference: OpenAI Batch API

    For bulk, non-real-time workloads (nightly reports, batch classification, bulk extraction), OpenAI's Batch API reduces cost by 50% with a 24-hour turnaround. Combined with prompt caching, the effective cost per token on stable system prompts can be reduced by up to 75%. The Batch API accepts JSONL files of up to 50,000 requests and processes them asynchronously.

    Key Takeaways

    • •Each provider handles TTL differently — understand your provider's model
    • •Monitor cache hit rate, cost savings, and latency improvements
    • •Build cost models to evaluate caching ROI for your usage pattern
    • •Keep caches warm during low-traffic periods if cost-effective
    0 of 5 lessons marked complete

    Progress is kept in this browser only.