45 min
Lesson 4 of 5
Production Cost Optimization
From the course Prompt Caching & Cost Optimization
Beyond Caching: Full Cost Optimization
Model Selection Optimization
The biggest cost lever is choosing the right model:
- Route simple tasks to smaller models (GPT-4o-mini, Haiku, Flash Lite)
- Reserve large models for complex reasoning only
- Build a router that classifies query complexity and selects models
Token Reduction Techniques
- Compress prompts: Remove redundant instructions, use concise language
- Truncate context: Only include relevant conversation history
- Structured outputs: Use JSON mode to eliminate verbose prose
- max_tokens: Always set to prevent runaway responses
Batching
For non-real-time workloads:
- OpenAI Batch API: 50% cost reduction, 24h turnaround
- Collect requests and submit in bulk
- Process results asynchronously
Architecture Patterns
Two-Stage Pipeline:
- Fast/cheap model classifies and extracts key info
- Expensive model only called when necessary
Semantic Caching:
- Cache responses by semantic similarity, not exact match
- Use embeddings to find similar previous queries
- Return cached responses for near-duplicate questions
Request Deduplication:
- Hash incoming requests
- Return cached responses for identical requests within a time window
- Particularly effective for autocomplete and search suggestions
Best Practice: Use Model Routing with a Lightweight Classifier
Route simple queries to cheap models automatically:
async function selectModel(query: string): Promise<string> {
const resp = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: `One word: is this "simple" (factual lookup) or "complex" (reasoning needed)?
"${query}"` }],
max_tokens: 5,
});
return resp.choices[0].message.content?.trim() === "simple" ? "gpt-4o-mini" : "gpt-4o";
}
Routing 70% of queries to gpt-4o-mini typically reduces costs by 60% with minimal quality impact on simple tasks.
Named Product Reference: Portkey AI for LLM Cost Management
Portkey is an LLM gateway providing unified cost tracking, automatic model fallback, and semantic caching across providers. It acts as a drop-in OpenAI SDK replacement and adds cost analytics, budgeting, and routing rules through configuration — reducing the code needed to manage AI spend across multiple applications.
Key Takeaways
- •Model selection is the biggest cost optimization lever
- •Build routers that classify complexity and select appropriate models
- •Use batching for 50% savings on non-urgent workloads
- •Semantic caching extends savings beyond exact prefix matching
0 of 5 lessons marked complete
Progress is kept in this browser only.