Chunking and retrieval are where most RAG quality issues originate. Optimizing these two steps can dramatically improve your system's accuracy.

Chunking Strategies Compared

1. Fixed-Size Chunking Split text every N tokens regardless of content.

• Pros: Simple, predictable chunk sizes
• Cons: Splits sentences and paragraphs mid-thought
• Best for: Homogeneous text (logs, transcripts)

2. Recursive Character Splitting Try splitting by paragraphs, then sentences, then words — stopping when chunks are the right size.

• Pros: Respects natural text boundaries
• Cons: Variable chunk sizes
• Best for: General documents, articles, documentation

3. Semantic Chunking Use an embedding model to detect topic shifts and split there.

• Pros: Each chunk covers one topic
• Cons: Slower, requires embedding calls during ingestion
• Best for: Long documents with multiple topics

4. Document Structure-Aware Split by headings, sections, and subsections in structured documents.

• Pros: Preserves document hierarchy and context
• Cons: Requires structured input (Markdown, HTML, DOCX)
• Best for: Technical docs, manuals, handbooks

5. Agentic Chunking Use an LLM to decide where to split based on content understanding.

• Pros: Highest quality splits
• Cons: Expensive and slow at ingestion time
• Best for: High-value documents where quality matters most

Retrieval Optimization Techniques

Hybrid Search Combine vector (semantic) search with keyword (BM25) search. This catches both meaning-based and exact-match queries.

Example: Searching for "GDPR Article 17" — vector search understands the topic, keyword search matches the exact article number.

Query Transformation Transform the user's question before searching:

• HyDE (Hypothetical Document Embeddings) — Generate a hypothetical answer, then search for documents similar to that answer
• Query decomposition — Break complex questions into sub-queries
• Query expansion — Add synonyms and related terms

Multi-Query Retrieval Generate 3-5 variations of the user's question and search for all of them. Merge and de-duplicate results.

Re-Ranking After initial retrieval, use a cross-encoder model to re-score results for relevance. This is more accurate than vector similarity alone but slower.

Popular re-rankers:

• Cohere Rerank
• BGE Reranker
• FlashRank (lightweight, open source)

Contextual Retrieval (Anthropic's Approach) Prepend each chunk with a brief context summary explaining where it fits in the larger document. This improves retrieval accuracy by 49% (per Anthropic's benchmarks).

Example:

Before: "Employees receive 20 days PTO per year..."
After: "[From: Company Handbook > Chapter 5: Benefits > Section 5.2: Time Off] Employees receive 20 days PTO per year..."

Evaluation Metrics

Track these to measure and improve retrieval quality:

• Recall@K — What percentage of relevant documents appear in the top K results?
• Precision@K — What percentage of the top K results are actually relevant?
• MRR (Mean Reciprocal Rank) — How high does the first relevant result appear?
• Answer correctness — Does the final generated answer match the ground truth?

Build an evaluation dataset of 50-100 question-answer pairs with their source documents. Run this after every change to your pipeline.