Production Scaling & RAG Integration
From the course Vector Search with Pinecone
Building Production RAG with Pinecone
RAG Pipeline
async function ragQuery(question: string) {
// 1. Embed the question
const queryVector = await embed([question]);
// 2. Retrieve relevant documents
const results = await index.query({
vector: queryVector[0],
topK: 5,
includeMetadata: true,
});
// 3. Build context from results
const context = results.matches
.map(m => m.metadata.text)
.join("\n\n");
// 4. Generate answer with context
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [
{ role: "system", content: `Answer based on this context:\n${context}` },
{ role: "user", content: question },
],
});
return { answer: response.choices[0].message.content, sources: results.matches };
}
Chunking Strategies
- Fixed-size: Split by token count (e.g., 512 tokens per chunk)
- Semantic: Split at paragraph or section boundaries
- Sliding window: Overlapping chunks for context preservation
- Recursive: Split large sections, then sub-split if still too large
Scaling Tips
- Monitor query latency percentiles (p50, p95, p99)
- Use pod-based indexes for consistent low-latency requirements
- Implement result caching for frequently asked questions
- Set up index replicas for read-heavy workloads
Common Pitfall: Over-Relying on topK Without Score Thresholding
Pinecone always returns exactly topK results even when none are truly relevant. Always filter by score threshold:
const MIN_SCORE = 0.75;
const filtered = results.matches.filter(m => (m.score ?? 0) >= MIN_SCORE);
if (filtered.length === 0) {
return "No relevant information found for this query.";
}
Best Practice: Monitor Retrieval Quality with Golden Queries
Maintain a small set of "golden queries" — questions where you know the correct answer and which document chunk should be retrieved. Run these after every embedding model or chunking change to catch retrieval regressions before they reach users:
const GOLDEN_QUERIES = [
{ query: "What is the refund policy?", expectedDocId: "policy-doc-refunds" },
{ query: "How do I reset my password?", expectedDocId: "help-password-reset" },
];
for (const gq of GOLDEN_QUERIES) {
const results = await ragQuery(gq.query);
const found = results.sources.some(s => s.id === gq.expectedDocId);
console.log(`[Golden] "${gq.query}": ${found ? "PASS" : "FAIL"}`);
}
Named Product Reference: Langfuse for RAG Observability
Langfuse is an open-source observability platform that traces each retrieval query — showing the query vector, retrieved chunks with scores, and the LLM answer — making it easy to identify whether poor answers are caused by bad retrieval or bad generation. It integrates with Pinecone, ChromaDB, and most vector databases through OpenTelemetry instrumentation.
Key Takeaways
- •RAG combines vector search retrieval with LLM generation
- •Chunking strategy significantly impacts retrieval quality
- •Monitor latency percentiles and cache frequent queries
- •Use index replicas for read-heavy production workloads
Progress is kept in this browser only.