Let's build a complete semantic search system from ingestion to query, covering chunking strategies, embedding generation, and retrieval optimization.

The Pipeline

1. Document loading → 2. Chunking → 3. Embedding → 4. Indexing → 5. Query → 6. Re-ranking

Step 1: Document Loading

Extract text from your source documents. Common formats:

• PDF: Use PyMuPDF or Unstructured for layout-aware extraction
• HTML: BeautifulSoup or Trafilatura for content extraction
• Markdown: Direct parsing with metadata preservation
• Databases: SQL queries with pagination

Step 2: Chunking Strategies

How you split documents dramatically affects retrieval quality:

• Fixed-size chunks (500-1000 tokens): Simple, predictable. Use overlapping windows (100-200 tokens) to avoid splitting mid-sentence.
• Semantic chunking: Split at natural boundaries (paragraphs, sections). Better coherence but variable sizes.
• Recursive character splitting: LangChain's default. Tries multiple separators (\n\n, \n, sentence, word) in order.
• Agentic chunking: Use an LLM to determine optimal split points. Highest quality but slowest.

Optimal chunk size depends on your embedding model's context window and your retrieval needs. Start with 512 tokens and experiment.

Step 3: Embedding Generation

from openai import OpenAI

client = OpenAI()

def embed_texts(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return [item.embedding for item in response.data]

Batch your embedding calls (up to 2048 texts per request for OpenAI) to minimize API calls.

Step 4-5: Indexing & Query

import chromadb

client = chromadb.Client()
collection = client.create_collection("docs")

# Index
collection.add(
    documents=chunks,
    ids=[f"chunk_{i}" for i in range(len(chunks))],
    metadatas=[{"source": "manual.pdf", "page": i} for i in range(len(chunks))],
)

# Query
results = collection.query(
    query_texts=["How do I configure authentication?"],
    n_results=5,
)

Step 6: Re-ranking

Initial vector search retrieves candidates; re-ranking improves precision:

• Cross-encoder re-ranking: A model that scores query-document pairs together. Much more accurate than bi-encoder similarity but too slow for initial retrieval.
• Cohere Rerank: Managed re-ranking API. Easy to integrate.
• ColBERT: Token-level similarity scoring. Excellent quality with reasonable speed.