Building an Embedding Pipeline

The Full Pipeline

1. Ingest — Collect documents from various sources
2. Chunk — Split documents into searchable segments
3. Embed — Convert chunks into vectors
4. Index — Store vectors in a vector database
5. Serve — Handle search queries in real-time

Chunking Strategies

Chunking is the most impactful decision in your pipeline:

Fixed-size chunking:

def chunk_fixed(text, chunk_size=512, overlap=50):
    tokens = tokenize(text)
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunks.append(detokenize(tokens[i:i + chunk_size]))
    return chunks

Semantic chunking:

• Split at paragraph/section boundaries
• Keep related content together
• Preserve headers and context

Recursive chunking:

• Try splitting by sections first
• Then paragraphs, then sentences
• Stop when chunks are target size

Embedding Model Selection

Consider these factors:

• Accuracy: How well does it capture semantic meaning?
• Speed: Latency per embedding (local vs API)
• Dimensions: Higher = more precise but more storage
• Cost: Free local models vs paid APIs
• Language: Multilingual support if needed

Metadata Enrichment

Attach metadata during indexing for filtering:

• Source document, URL, title
• Creation/modification dates
• Categories, tags, authors
• Chunk position (start, middle, end of document)

Common Pitfall: Choosing Chunk Size Without Testing

Default chunk sizes (512 tokens) are not optimal for all document types. Technical documentation needs smaller chunks (128–256 tokens) so each chunk covers one concept. Legal contracts may need larger chunks (1024 tokens) to preserve clause context. Test at least three chunk sizes on a representative sample of your documents before committing to a production ingestion:

# Quick chunk-size experiment framework
for chunk_size in [128, 256, 512, 1024]:
    chunks = chunk_documents(docs, chunk_size=chunk_size, overlap=chunk_size // 8)
    scores = evaluate_retrieval(chunks, test_queries)
    print(f"Chunk {chunk_size}: P@5={scores['precision_at_5']:.3f}")

Best Practice: Prepend Document Title to Each Chunk

Embedding a chunk without its surrounding context often produces poor vectors — the model cannot tell whether "the patient was treated with X" is from a medical research paper or a legal complaint. Prepend the document title and section heading to each chunk before embedding:

def enrich_chunk(chunk: str, title: str, section: str) -> str:
    return f"Document: {title}
Section: {section}

{chunk}"

This technique, called "contextual retrieval" (popularized by Anthropic), typically improves retrieval accuracy by 35–50%.

Named Product Reference: LlamaIndex for Ingestion Pipelines

LlamaIndex (formerly GPT Index) provides a high-level ingestion pipeline with built-in support for chunking strategies, metadata enrichment, and multiple vector store backends. The IngestionPipeline class handles chunking, embedding, and upserting in a single configurable object. For teams building complex ingestion pipelines, LlamaIndex reduces boilerplate code significantly compared to building the pipeline from scratch.