Implementation Strategies
From the course Prompt Caching & Cost Optimization
Caching Strategies by Use Case
System Prompt Caching
The most common pattern — cache your system instructions:
// Structure messages for optimal caching
const messages = [
// CACHED: System prompt (stable across all requests)
{ role: "system", content: longSystemPrompt },
// CACHED: Few-shot examples (stable)
{ role: "user", content: "Example input 1" },
{ role: "assistant", content: "Example output 1" },
// NOT CACHED: User's actual query (changes each time)
{ role: "user", content: userQuery },
];
Document Context Caching
For Q&A over large documents:
- Place the document content in the system prompt or first messages
- Mark it as cacheable
- Ask different questions in subsequent requests
- Each question reuses the cached document processing
Conversation Caching
Multi-turn conversations benefit from automatic prefix caching:
- Turn 1: System + User message (no cache)
- Turn 2: System + User 1 + Assistant 1 + User 2 (cache hit on prefix)
- Turn 3: Even more prefix reuse
Anti-Patterns to Avoid
- Don't put dynamic content (timestamps, random IDs) in cached prefixes
- Don't reorder messages between requests
- Don't exceed TTL without sending keep-alive requests
- Don't cache very short prompts — overhead exceeds savings
Common Pitfall: Exceeding the Cache Minimum Token Threshold
Anthropic requires at least 1,024 tokens (for Haiku) or 2,048 tokens (for Sonnet/Opus) in the cached prefix before caching is applied. Many teams enable caching for a 500-token system prompt and wonder why there are no cache hits — the prompt is too short. Verify the cached content exceeds the minimum threshold:
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
// Verify: system prompt must be >2048 tokens for Sonnet to be cached
const systemPromptTokens = await client.messages.countTokens({
model: "claude-sonnet-4-20250514",
system: [{ type: "text", text: systemPrompt, cache_control: { type: "ephemeral" } }],
messages: [],
});
console.log("System prompt tokens:", systemPromptTokens.input_tokens);
// Must be >2048 for caching to activate
Best Practice: Use Few-Shot Examples as Cacheable Content
Few-shot examples (showing the model the desired input/output format 3–10 times) significantly improve output quality but add tokens to every request. Cache the examples to get quality improvement at minimal ongoing cost:
const messages = [
// Cache the system prompt + all examples together
{ role: "user", content: [
{ type: "text", text: systemPrompt + "
" + fewShotExamples, cache_control: { type: "ephemeral" } }
]},
{ role: "assistant", content: "Understood." },
// Uncached: the actual user query
{ role: "user", content: userQuery },
];
Named Product Reference: Redis for Semantic Caching Layer
Beyond provider-level prompt caching, teams implement a semantic caching layer using Redis and an embedding model. Incoming queries are embedded, compared against cached query embeddings, and if sufficiently similar (cosine similarity > 0.95), the cached response is returned directly without any LLM call. Tools like GPTCache or Zep's semantic cache provide this out-of-the-box.
Key Takeaways
- •Structure messages with static content first for optimal caching
- •Document Q&A benefits enormously from context caching
- •Multi-turn conversations get increasing cache benefits over time
- •Avoid dynamic content in cached prefixes
Progress is kept in this browser only.