What Actually Breaks in a Production RAG Pipeline

The demo version of retrieval-augmented generation takes an afternoon. You chunk some documents, embed them, drop them in a vector store, and stuff the top five results into a prompt. It works impressively well on the twenty documents you tested with.

The production version is a different system, and almost all of the difficulty sits upstream of the model.

Retrieval is the bottleneck, not generation

When a RAG answer is wrong, the instinct is to blame the model or the prompt. In practice, the retrieved context usually did not contain the answer. No amount of prompt engineering recovers from that.

The diagnostic is cheap and worth building on day one: log the retrieved chunks alongside every answer, then sample failures and ask a single question — was the answer present in the retrieved context?

• Answer present, response wrong → generation problem. Prompt, model, or context ordering.
• Answer absent → retrieval problem. Everything below.

In my experience the second bucket dominates, often heavily.

Chunking decides your ceiling

Chunking is usually treated as a preprocessing detail and set to whatever the tutorial used. It determines the upper bound on retrieval quality.

Fixed-size character chunking splits tables down the middle, severs a heading from the paragraph it introduces, and produces chunks that are individually meaningless. An embedding of a meaningless chunk is a meaningless vector.

def chunk_document(doc, target_tokens=512, overlap_tokens=64):
    """Split on structure first, size second.

    Sections that fit stay whole. Only oversized sections get split, and
    those carry an overlap so a sentence spanning the boundary is
    retrievable from either side.
    """
    chunks = []
    for section in doc.sections:          # headings, list blocks, tables
        if section.token_count <= target_tokens:
            chunks.append(section)
            continue
        chunks.extend(
            split_with_overlap(section, target_tokens, overlap_tokens)
        )
    return chunks

Two rules carry most of the benefit: split on document structure before falling back to size, and prepend the section heading to every chunk so an isolated paragraph still carries its context into the embedding.

Metadata filtering beats a bigger k

The reflex when retrieval misses is to raise k. This dilutes the context window with near-misses and makes the generation step harder.

Filtering on metadata is usually the better lever:

results = index.query(
    vector=embed(question),
    top_k=8,
    filter={
        "doc_type": {"$in": ["policy", "endorsement"]},
        "effective_date": {"$gte": as_of_date},
        "business_unit": user.business_unit,
    },
)

For enterprise corpora this matters more than in general search, because the corpus contains many near-identical documents that differ only by version, region or effective date. Semantic similarity cannot distinguish the 2024 policy wording from the 2025 one. Metadata can.

Most enterprise retrieval failures I’ve traced were not “the model didn’t understand.” They were “we retrieved the superseded version of the right document.”

Re-embedding is a migration, not a config change

Changing your embedding model invalidates every vector in the index. This is obvious stated plainly and routinely forgotten in planning.

Treat it as a data migration with the usual apparatus:

1. Version the index by embedding model, never overwrite in place
2. Build the new index alongside the old one
3. Compare retrieval quality on a fixed evaluation set before cutting over
4. Keep the old index until the new one has been live long enough to trust

You need that fixed evaluation set — a few hundred question-and-expected-source pairs — before you need it. Building one after retrieval quality regresses means you cannot tell whether it regressed.

Where cost accumulates

Content-hash caching on embeddings is the highest-leverage item on that list. Enterprise corpora re-ingest largely unchanged documents constantly, and hashing chunk content before embedding removes most of the repeat cost for a few lines of code.

What I’d build first next time

• The retrieval log, before anything else — you cannot debug what you cannot see
• A fixed evaluation set of question-to-source pairs, however small
• Content-hash caching in the embedding step
• Structure-aware chunking rather than fixed-size
• Metadata filters, designed alongside the ingestion schema rather than bolted on afterwards

The generation layer is the part everyone demos. The retrieval layer is the part that determines whether the demo survives contact with a real corpus.