Measuring Search Quality

Key Metrics

• Precision@K: Fraction of top-K results that are relevant
• Recall@K: Fraction of all relevant docs found in top-K
• MRR (Mean Reciprocal Rank): Average of 1/rank of first relevant result
• NDCG: Normalized Discounted Cumulative Gain — considers graded relevance

Building an Evaluation Dataset

{
  "queries": [
    {
      "query": "How to deploy a React app",
      "relevant_doc_ids": ["doc-42", "doc-87", "doc-103"],
      "irrelevant_doc_ids": ["doc-5", "doc-200"]
    }
  ]
}

A/B Testing Search Changes

Test changes systematically:

1. Define your success metric (click-through rate, task completion)
2. Run the old and new search systems in parallel
3. Randomly assign users to each system
4. Compare metrics with statistical significance

Common Optimization Levers

• Chunking: Adjust size, overlap, and strategy
• Embedding model: Try different models and dimensions
• Reranking: Add or swap rerankers
• Metadata filters: Pre-filter to reduce noise
• Prompt tuning: Optimize query prompts for the embedding model

Debugging Poor Results

• Check if the relevant document exists in the index
• Verify embedding quality by inspecting nearest neighbors
• Look for chunking issues splitting relevant content
• Test with the exact document text as a query (should score ~1.0)

Common Pitfall: Treating All Evaluation Metrics as Equal

NDCG and MRR measure very different things. MRR measures whether the first relevant result appears early — critical for single-answer retrieval. NDCG measures the overall ranking quality across all relevant results — important for RAG where multiple chunks contribute to the answer. Choose your primary metric based on your use case before starting evaluation.

Best Practice: Use Synthetic Evaluation Data for Cold-Start Evaluation

If you don't have labeled test queries, generate them synthetically. Use an LLM to create 3–5 questions that each document chunk could answer:

for chunk in chunks:
    questions = llm.generate(
        f"Generate 3 questions that are answered by this text:

{chunk.text}"
    )
    for q in questions:
        eval_dataset.append({"query": q, "relevant_doc_id": chunk.id})

This technique generates hundreds of labeled query-document pairs in minutes. The synthetic labels are noisy but sufficient for catching major retrieval regressions.

Named Product Reference: RAGAS for Automated RAG Evaluation

RAGAS (RAG Assessment) is an open-source framework that evaluates RAG pipelines using metrics like faithfulness (does the answer follow from the retrieved context?), answer relevancy, and context precision. It uses an LLM-as-judge approach and generates a composite score that correlates well with human evaluations. Install with pip install ragas and integrate with LangChain, LlamaIndex, or any custom RAG pipeline.