Understanding Prompt Caching
From the course Prompt Caching & Cost Optimization
What Is Prompt Caching?
Prompt caching stores processed prompt prefixes so they don't need to be re-computed on subsequent requests, dramatically reducing costs and latency.
Why Caching Matters
AI API costs scale with token count. Common patterns like system prompts, few-shot examples, and document context are repeated across requests — caching eliminates this redundancy.
How It Works
- First request: Full prompt is processed and cached
- Subsequent requests: Cached prefix is reused, only new tokens are processed
- Cache hit: Matching prefix found → reduced cost and latency
- Cache miss: No match → full processing (cache is created)
Provider Comparison
| Provider | Cache Type | Min Tokens | TTL | Savings |
|---|---|---|---|---|
| Anthropic | Explicit (opt-in) | 1,024-2,048 | 5 min (refreshing) | Up to 90% |
| OpenAI | Automatic | 1,024 | ~5-10 min | 50% on cached |
| Explicit | 32,768 | Custom | Varies by model |
Key Concepts
- Prefix matching: Cache only works for exact prefix matches
- Cache writes: First-time caching has a small overhead cost
- Cache reads: Subsequent hits are significantly cheaper
- Ordering matters: Rearranging content breaks the cache
Common Pitfall: Putting Variable Content Before Stable Content
Cache prefix matching requires that the cached portion appears at the beginning of the prompt. If you put dynamic content (the user's message) before static content (system instructions, few-shot examples), you break the cache on every request. Always structure prompts with the most stable content first:
[CACHED] System prompt (stable across all users)
[CACHED] Few-shot examples (stable)
[CACHED] Document context (stable per document)
[NOT CACHED] User's current message (changes every request)
Best Practice: Check Cache Hit Indicators in Responses
Both Anthropic and OpenAI return cache usage metrics in their API responses. Log these to verify your caching strategy is working:
// Anthropic
const usage = response.usage;
console.log("Cache read tokens:", usage.cache_read_input_tokens); // Tokens from cache
console.log("Cache write tokens:", usage.cache_creation_input_tokens); // Tokens cached this call
// OpenAI
const usage = response.usage;
console.log("Cached tokens:", usage.prompt_tokens_details?.cached_tokens);
If cache_read_input_tokens is 0 on requests you expect to hit the cache, your prompt structure has a problem.
Named Product Reference: Anthropic's Claude Prompt Caching
Anthropic's explicit caching (activated by adding cache_control: { type: "ephemeral" } to message parts) offers up to 90% cost reduction on cached tokens and 85% latency reduction. This is the most aggressive caching available from any major provider. For chatbots with large system prompts or knowledge bases, enabling prompt caching typically reduces costs by 50-70% in production.
Key Takeaways
- •Prompt caching stores processed prefixes to avoid recomputation
- •Savings range from 50-90% depending on the provider
- •Caches require exact prefix matches — ordering matters
- •Different providers offer automatic vs. explicit caching
Progress is kept in this browser only.