AI Model Wiki

    Skip to main content
    40 min
    Lesson 1 of 5

    Understanding Prompt Caching

    From the course Prompt Caching & Cost Optimization

    What Is Prompt Caching?

    Prompt caching stores processed prompt prefixes so they don't need to be re-computed on subsequent requests, dramatically reducing costs and latency.

    Why Caching Matters

    AI API costs scale with token count. Common patterns like system prompts, few-shot examples, and document context are repeated across requests — caching eliminates this redundancy.

    How It Works

    1. First request: Full prompt is processed and cached
    2. Subsequent requests: Cached prefix is reused, only new tokens are processed
    3. Cache hit: Matching prefix found → reduced cost and latency
    4. Cache miss: No match → full processing (cache is created)

    Provider Comparison

    ProviderCache TypeMin TokensTTLSavings
    AnthropicExplicit (opt-in)1,024-2,0485 min (refreshing)Up to 90%
    OpenAIAutomatic1,024~5-10 min50% on cached
    GoogleExplicit32,768CustomVaries by model

    Key Concepts

    • Prefix matching: Cache only works for exact prefix matches
    • Cache writes: First-time caching has a small overhead cost
    • Cache reads: Subsequent hits are significantly cheaper
    • Ordering matters: Rearranging content breaks the cache

    Common Pitfall: Putting Variable Content Before Stable Content

    Cache prefix matching requires that the cached portion appears at the beginning of the prompt. If you put dynamic content (the user's message) before static content (system instructions, few-shot examples), you break the cache on every request. Always structure prompts with the most stable content first:

    [CACHED] System prompt (stable across all users)
    [CACHED] Few-shot examples (stable)
    [CACHED] Document context (stable per document)
    [NOT CACHED] User's current message (changes every request)
    

    Best Practice: Check Cache Hit Indicators in Responses

    Both Anthropic and OpenAI return cache usage metrics in their API responses. Log these to verify your caching strategy is working:

    // Anthropic
    const usage = response.usage;
    console.log("Cache read tokens:", usage.cache_read_input_tokens);    // Tokens from cache
    console.log("Cache write tokens:", usage.cache_creation_input_tokens); // Tokens cached this call
    
    // OpenAI
    const usage = response.usage;
    console.log("Cached tokens:", usage.prompt_tokens_details?.cached_tokens);
    

    If cache_read_input_tokens is 0 on requests you expect to hit the cache, your prompt structure has a problem.

    Named Product Reference: Anthropic's Claude Prompt Caching

    Anthropic's explicit caching (activated by adding cache_control: { type: "ephemeral" } to message parts) offers up to 90% cost reduction on cached tokens and 85% latency reduction. This is the most aggressive caching available from any major provider. For chatbots with large system prompts or knowledge bases, enabling prompt caching typically reduces costs by 50-70% in production.

    Key Takeaways

    • •Prompt caching stores processed prefixes to avoid recomputation
    • •Savings range from 50-90% depending on the provider
    • •Caches require exact prefix matches — ordering matters
    • •Different providers offer automatic vs. explicit caching
    0 of 5 lessons marked complete

    Progress is kept in this browser only.