AI Model Wiki

    Skip to main content
    45 min
    Lesson 4 of 5

    Production Scaling & RAG Integration

    From the course Vector Search with Pinecone

    Building Production RAG with Pinecone

    RAG Pipeline

    async function ragQuery(question: string) {
      // 1. Embed the question
      const queryVector = await embed([question]);
    
      // 2. Retrieve relevant documents
      const results = await index.query({
        vector: queryVector[0],
        topK: 5,
        includeMetadata: true,
      });
    
      // 3. Build context from results
      const context = results.matches
        .map(m => m.metadata.text)
        .join("\n\n");
    
      // 4. Generate answer with context
      const response = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [
          { role: "system", content: `Answer based on this context:\n${context}` },
          { role: "user", content: question },
        ],
      });
    
      return { answer: response.choices[0].message.content, sources: results.matches };
    }
    

    Chunking Strategies

    • Fixed-size: Split by token count (e.g., 512 tokens per chunk)
    • Semantic: Split at paragraph or section boundaries
    • Sliding window: Overlapping chunks for context preservation
    • Recursive: Split large sections, then sub-split if still too large

    Scaling Tips

    • Monitor query latency percentiles (p50, p95, p99)
    • Use pod-based indexes for consistent low-latency requirements
    • Implement result caching for frequently asked questions
    • Set up index replicas for read-heavy workloads

    Common Pitfall: Over-Relying on topK Without Score Thresholding

    Pinecone always returns exactly topK results even when none are truly relevant. Always filter by score threshold:

    const MIN_SCORE = 0.75;
    const filtered = results.matches.filter(m => (m.score ?? 0) >= MIN_SCORE);
    if (filtered.length === 0) {
      return "No relevant information found for this query.";
    }
    

    Best Practice: Monitor Retrieval Quality with Golden Queries

    Maintain a small set of "golden queries" — questions where you know the correct answer and which document chunk should be retrieved. Run these after every embedding model or chunking change to catch retrieval regressions before they reach users:

    const GOLDEN_QUERIES = [
      { query: "What is the refund policy?", expectedDocId: "policy-doc-refunds" },
      { query: "How do I reset my password?", expectedDocId: "help-password-reset" },
    ];
    
    for (const gq of GOLDEN_QUERIES) {
      const results = await ragQuery(gq.query);
      const found = results.sources.some(s => s.id === gq.expectedDocId);
      console.log(`[Golden] "${gq.query}": ${found ? "PASS" : "FAIL"}`);
    }
    

    Named Product Reference: Langfuse for RAG Observability

    Langfuse is an open-source observability platform that traces each retrieval query — showing the query vector, retrieved chunks with scores, and the LLM answer — making it easy to identify whether poor answers are caused by bad retrieval or bad generation. It integrates with Pinecone, ChromaDB, and most vector databases through OpenTelemetry instrumentation.

    Key Takeaways

    • •RAG combines vector search retrieval with LLM generation
    • •Chunking strategy significantly impacts retrieval quality
    • •Monitor latency percentiles and cache frequent queries
    • •Use index replicas for read-heavy production workloads
    0 of 5 lessons marked complete

    Progress is kept in this browser only.