Most retrieval-augmented generation projects start the same way. Someone loads a few hundred PDFs into a vector database, wires up an embedding model and a chat endpoint, and asks it a question in a meeting. The answer is good. Everyone nods. Six weeks later the same system is answering a customer about a refund policy that was replaced in March, quoting a contract the user was never supposed to see, and taking nine seconds to do it.
None of that is a model problem. It is a retrieval, data and operations problem, and it is where almost all of the engineering effort in RAG in production actually goes. This is the checklist we work through when a client asks us to take a prototype live.
Chunking is a data modeling decision
Fixed 512-token windows with a bit of overlap are fine for a demo. They fall apart on real documents because they cut tables in half, separate a heading from the paragraph it governs, and merge two unrelated sections of a policy into one embedding. The retriever then returns a chunk that is half relevant, and the model fills the other half with guesses.
Chunk along the structure the document already has. For HTML and Markdown that means headings. For PDFs it usually means running a layout parser first so you know where sections, lists and tables begin. Keep tables whole, even if that makes the chunk long. Prepend the document title and the heading path to every chunk before embedding it, so a chunk that says "this limit is 30 days" also carries "Returns Policy > EU customers". That one change often does more for answer quality than swapping embedding models.
Store metadata alongside each chunk from day one: source ID, version, last-modified date, language, and the access groups allowed to read it. You will need every one of those fields later, and backfilling them across a few million chunks is miserable.
Hybrid search and reranking
Dense embeddings are good at meaning and bad at exact strings. Ask a pure vector index for "error E-4031" or a SKU like "AX-220-B" and it will happily return chunks about similar-sounding errors. Users in support, legal and engineering ask exactly these kinds of questions all day.
The fix is hybrid retrieval: run BM25 (or whatever keyword index your search engine provides) next to the vector search and merge the two lists, typically with reciprocal rank fusion. Most mature search engines and several vector databases now support this natively, so it is rarely a large build.
Then add a reranker. Retrieve something like 50 candidates cheaply, score them with a cross-encoder reranking model against the actual question, and pass the top 5 to 8 to the LLM. Cross-encoders read the query and the chunk together, so they catch relevance that a single embedding comparison misses. The cost is latency, usually tens to a couple of hundred milliseconds depending on the model and where it runs. In our experience it is the single most reliable quality improvement after fixing chunking.
Build the evaluation set before you tune anything
Without an evaluation set, every change is a vibe check. You swap the embedding model, ask three questions, and the answers look about the same. You have learned nothing.
Collect 150 to 300 real questions. Pull them from support tickets, search logs, or by sitting with the people who will use the system. For each one, record which documents contain the answer and, ideally, a short reference answer written by a subject-matter expert. Include questions with no answer in the corpus, because a good system should say so instead of improvising.
Measure the two halves separately. For retrieval, track recall@k: did the right document appear in the top k results? For generation, check whether the answer is correct, whether it is grounded in the retrieved text, and whether it cites the right source. An LLM grader can handle the generation scoring at volume, provided you spot-check it against human labels first. Run the whole set on every change to chunking, prompts, models or ranking. It takes minutes and it ends arguments.
Citations, freshness and re-indexing
Make the model cite chunk IDs inline, then render those as links to the source document and section. Validate the citations after generation: if the model cites a chunk that was not in its context, or makes a claim with no citation at all, flag it or drop the sentence. Citations are what let a user trust an answer in ten seconds instead of re-reading the policy themselves.
Freshness is the failure that embarrasses you in front of customers. Treat the index as a derived dataset with a pipeline, not a one-off upload. Hook into the source systems (SharePoint, Confluence, Google Drive, your CMS) through change events or scheduled diffs, re-chunk and re-embed only what changed, and hard-delete chunks from removed or superseded documents. Keep a version field so you can tell which copy of a document a given answer came from when someone files a complaint.
Plan for full re-embeds too. Changing your embedding model means re-indexing the entire corpus, so build the pipeline to write to a new index and switch over with an alias rather than mutating the live one.
Access control has to happen at retrieval time
The tempting shortcut is to retrieve everything and ask the model not to mention restricted content. Do not do this. Anything placed in the context window can end up in the answer, and prompt injection makes "please don't reveal" instructions unreliable.
Instead, carry the source system's permissions onto each chunk as access-group metadata, resolve the user's groups at query time, and apply them as a filter inside the search query itself. Sync permission changes on the same pipeline as content changes, because a revoked share that lingers in your index for a week is a real incident. Test it with a handful of seeded documents only one test account can see, and make that test part of CI.
Cost and latency budgets
Write the budget down before launch. A typical target for an internal assistant looks something like this:
| Stage | Typical latency budget | Main lever |
|---|---|---|
| Query rewrite / embedding | 50–150 ms | Smaller embedding model, cache common queries |
| Hybrid retrieval | 50–200 ms | Index tuning, metadata pre-filters |
| Reranking | 50–300 ms | Fewer candidates, smaller cross-encoder |
| LLM generation | 1–4 s to full answer | Shorter context, smaller model, streaming |
Context size drives most of the cost. Sending 8 tight, reranked chunks instead of 30 loose ones cuts input tokens dramatically and usually improves answers too, since the model has less noise to ignore. Stream the response so users see text within a second. Route simple lookups to a cheaper model and reserve the large one for questions that need synthesis across documents. Track cost per answered question, not just total spend, so you notice when a prompt change quietly doubles it.
Failure modes to design for
- Nothing relevant retrieved: set a reranker score threshold and answer "I couldn't find this in the documents" instead of letting the model guess.
- Conflicting sources: two versions of a policy both retrieved. Prefer the newest by metadata and show both citations when it matters.
- Prompt injection inside documents: a shared file containing instructions aimed at the model. Treat retrieved text as data, strip it of tool-calling power, and log suspicious patterns.
- Stale index: monitor the lag between a source change and its re-index, and alert when it exceeds your agreed window.
- Silent quality drift: sample production questions weekly, grade them, and add the failures to the evaluation set.
None of these are exotic. They show up in the first month of real traffic, every time. Teams that budget for them ship a system people keep using. Teams that don't end up with a clever demo that the support desk quietly stops opening.
Questions we get asked
- How many documents can a production RAG system handle?
- Corpus size is rarely the limit; modern search engines and vector databases handle millions of chunks. The harder limits are keeping that corpus fresh, permissioned and evaluated as it grows.
- Do we need to fine-tune a model for RAG?
- Usually not at first. Better chunking, hybrid search and reranking fix most quality problems. Fine-tuning becomes worth considering for domain-specific embeddings or strict output formats once retrieval is solid.
- How big should a RAG evaluation set be?
- Start with 150 to 300 real questions with known source documents, including some that have no answer in the corpus. Grow it by adding every production failure you find.
- Can RAG respect existing SharePoint or Google Drive permissions?
- Yes, if you sync access groups onto each chunk and filter at query time. Never rely on the prompt to hide restricted content from the model.