AI Model Wiki

    Skip to main content
    45 min
    Lesson 2 of 5

    Implementation Strategies

    From the course Prompt Caching & Cost Optimization

    Caching Strategies by Use Case

    System Prompt Caching

    The most common pattern — cache your system instructions:

    // Structure messages for optimal caching
    const messages = [
      // CACHED: System prompt (stable across all requests)
      { role: "system", content: longSystemPrompt },
      // CACHED: Few-shot examples (stable)
      { role: "user", content: "Example input 1" },
      { role: "assistant", content: "Example output 1" },
      // NOT CACHED: User's actual query (changes each time)
      { role: "user", content: userQuery },
    ];
    

    Document Context Caching

    For Q&A over large documents:

    1. Place the document content in the system prompt or first messages
    2. Mark it as cacheable
    3. Ask different questions in subsequent requests
    4. Each question reuses the cached document processing

    Conversation Caching

    Multi-turn conversations benefit from automatic prefix caching:

    • Turn 1: System + User message (no cache)
    • Turn 2: System + User 1 + Assistant 1 + User 2 (cache hit on prefix)
    • Turn 3: Even more prefix reuse

    Anti-Patterns to Avoid

    • Don't put dynamic content (timestamps, random IDs) in cached prefixes
    • Don't reorder messages between requests
    • Don't exceed TTL without sending keep-alive requests
    • Don't cache very short prompts — overhead exceeds savings

    Common Pitfall: Exceeding the Cache Minimum Token Threshold

    Anthropic requires at least 1,024 tokens (for Haiku) or 2,048 tokens (for Sonnet/Opus) in the cached prefix before caching is applied. Many teams enable caching for a 500-token system prompt and wonder why there are no cache hits — the prompt is too short. Verify the cached content exceeds the minimum threshold:

    import Anthropic from "@anthropic-ai/sdk";
    const client = new Anthropic();
    
    // Verify: system prompt must be >2048 tokens for Sonnet to be cached
    const systemPromptTokens = await client.messages.countTokens({
      model: "claude-sonnet-4-20250514",
      system: [{ type: "text", text: systemPrompt, cache_control: { type: "ephemeral" } }],
      messages: [],
    });
    console.log("System prompt tokens:", systemPromptTokens.input_tokens);
    // Must be >2048 for caching to activate
    

    Best Practice: Use Few-Shot Examples as Cacheable Content

    Few-shot examples (showing the model the desired input/output format 3–10 times) significantly improve output quality but add tokens to every request. Cache the examples to get quality improvement at minimal ongoing cost:

    const messages = [
      // Cache the system prompt + all examples together
      { role: "user", content: [
        { type: "text", text: systemPrompt + "
    
    " + fewShotExamples, cache_control: { type: "ephemeral" } }
      ]},
      { role: "assistant", content: "Understood." },
      // Uncached: the actual user query
      { role: "user", content: userQuery },
    ];
    

    Named Product Reference: Redis for Semantic Caching Layer

    Beyond provider-level prompt caching, teams implement a semantic caching layer using Redis and an embedding model. Incoming queries are embedded, compared against cached query embeddings, and if sufficiently similar (cosine similarity > 0.95), the cached response is returned directly without any LLM call. Tools like GPTCache or Zep's semantic cache provide this out-of-the-box.

    Key Takeaways

    • •Structure messages with static content first for optimal caching
    • •Document Q&A benefits enormously from context caching
    • •Multi-turn conversations get increasing cache benefits over time
    • •Avoid dynamic content in cached prefixes
    0 of 5 lessons marked complete

    Progress is kept in this browser only.