AI Model Wiki

    Skip to main content
    45 min
    Lesson 4 of 5

    Production Cost Optimization

    From the course Prompt Caching & Cost Optimization

    Beyond Caching: Full Cost Optimization

    Model Selection Optimization

    The biggest cost lever is choosing the right model:

    • Route simple tasks to smaller models (GPT-4o-mini, Haiku, Flash Lite)
    • Reserve large models for complex reasoning only
    • Build a router that classifies query complexity and selects models

    Token Reduction Techniques

    • Compress prompts: Remove redundant instructions, use concise language
    • Truncate context: Only include relevant conversation history
    • Structured outputs: Use JSON mode to eliminate verbose prose
    • max_tokens: Always set to prevent runaway responses

    Batching

    For non-real-time workloads:

    • OpenAI Batch API: 50% cost reduction, 24h turnaround
    • Collect requests and submit in bulk
    • Process results asynchronously

    Architecture Patterns

    Two-Stage Pipeline:

    1. Fast/cheap model classifies and extracts key info
    2. Expensive model only called when necessary

    Semantic Caching:

    • Cache responses by semantic similarity, not exact match
    • Use embeddings to find similar previous queries
    • Return cached responses for near-duplicate questions

    Request Deduplication:

    • Hash incoming requests
    • Return cached responses for identical requests within a time window
    • Particularly effective for autocomplete and search suggestions

    Best Practice: Use Model Routing with a Lightweight Classifier

    Route simple queries to cheap models automatically:

    async function selectModel(query: string): Promise<string> {
      const resp = await openai.chat.completions.create({
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: `One word: is this "simple" (factual lookup) or "complex" (reasoning needed)?
    "${query}"` }],
        max_tokens: 5,
      });
      return resp.choices[0].message.content?.trim() === "simple" ? "gpt-4o-mini" : "gpt-4o";
    }
    

    Routing 70% of queries to gpt-4o-mini typically reduces costs by 60% with minimal quality impact on simple tasks.

    Named Product Reference: Portkey AI for LLM Cost Management

    Portkey is an LLM gateway providing unified cost tracking, automatic model fallback, and semantic caching across providers. It acts as a drop-in OpenAI SDK replacement and adds cost analytics, budgeting, and routing rules through configuration — reducing the code needed to manage AI spend across multiple applications.

    Key Takeaways

    • •Model selection is the biggest cost optimization lever
    • •Build routers that classify complexity and select appropriate models
    • •Use batching for 50% savings on non-urgent workloads
    • •Semantic caching extends savings beyond exact prefix matching
    0 of 5 lessons marked complete

    Progress is kept in this browser only.