TTL Management & Monitoring
From the course Prompt Caching & Cost Optimization
Managing Cache Lifetime
TTL Strategies
Anthropic (5-minute refreshing):
- Cache refreshes on each hit — stays alive as long as you use it
- For bursty traffic, send periodic requests to keep cache warm
- For low-traffic, accept cache misses and rebuild
Google Gemini (custom TTL):
- Set explicit TTL from minutes to hours
- Longer TTL = higher storage costs but fewer cache misses
- Delete caches when no longer needed
OpenAI (automatic):
- No TTL management needed — fully automatic
- Caching happens transparently for prompts >1,024 tokens
- Check usage API for cache hit statistics
Monitoring Cache Performance
Track these metrics:
- Cache hit rate: Percentage of requests that hit the cache
- Cost savings: Compare actual spend vs. uncached estimate
- Latency improvement: Time to first token with vs. without cache
- Cache waste: Tokens cached but rarely reused
Cost Modeling
Build a simple model to evaluate caching ROI:
monthly_savings = (cached_tokens × cache_read_discount × request_count)
- (cached_tokens × cache_write_premium × cache_creates)
- (storage_cost × hours_cached)
If monthly_savings > 0, caching is worth it.
Common Pitfall: Not Accounting for Cache Write Cost
Caching is not free — providers charge a "cache write" premium (Anthropic charges 25% more than the normal input token price to create a cache entry). If you create a new cache entry on every request (because your cached prefix keeps changing), you pay the write premium on every call without getting the read discount. Monitor the ratio of cache writes to reads:
const metrics = {
cache_writes: response.usage.cache_creation_input_tokens,
cache_reads: response.usage.cache_read_input_tokens,
};
const ratio = metrics.cache_reads / (metrics.cache_writes + metrics.cache_reads);
console.log(`Cache efficiency: ${(ratio * 100).toFixed(1)}%`);
// Target: >80% reads, <20% writes for positive ROI
Best Practice: Combine Prompt Caching with Response Caching
For frequently asked questions, add a second caching layer that stores the final LLM response. If a user asks "What is your return policy?" for the tenth time today, return the cached answer directly — no LLM call at all. Hash the user's message (after normalization) as the cache key:
const cacheKey = `response:${hashQuery(userMessage)}`;
const cached = await redis.get(cacheKey);
if (cached) return JSON.parse(cached);
const response = await callLLM(systemPrompt, userMessage);
await redis.setex(cacheKey, 3600, JSON.stringify(response));
return response;
Named Product Reference: OpenAI Batch API
For bulk, non-real-time workloads (nightly reports, batch classification, bulk extraction), OpenAI's Batch API reduces cost by 50% with a 24-hour turnaround. Combined with prompt caching, the effective cost per token on stable system prompts can be reduced by up to 75%. The Batch API accepts JSONL files of up to 50,000 requests and processes them asynchronously.
Key Takeaways
- •Each provider handles TTL differently — understand your provider's model
- •Monitor cache hit rate, cost savings, and latency improvements
- •Build cost models to evaluate caching ROI for your usage pattern
- •Keep caches warm during low-traffic periods if cost-effective
Progress is kept in this browser only.