Chunking is an information design decision A RAG assistant cannot answer from a document it fails to retrieve. Chunking turns source material into passages...
Chunking is an information design decision
A RAG assistant cannot answer from a document it fails to retrieve. Chunking turns source material into passages that a search system can rank. The right boundary depends on the source: a policy section, API method, troubleshooting step, and table row do not share an ideal fixed length.
Start by preserving document structure. Keep headings with the paragraphs they introduce, retain list context, and record the source URL, version, and passage position. A passage without provenance may look relevant but cannot support a useful citation.
Build a representative question set
Collect actual questions from users or support staff. Include exact identifier lookups, conceptual questions, multi step procedures, ambiguous questions, and questions outside the approved collection. For each answerable question, mark one or more passages that should be retrievable. This set becomes the comparison point when changing chunk size, embeddings, or ranking.
Do not tune only against easy examples written by the same person who built the index. Hold back a test set and include documents that use similar words for different meanings.
Compare chunk strategies
A fixed character split is simple but can separate a condition from the rule it qualifies. A structure aware split keeps section headings and related paragraphs together. Overlap can preserve context across boundaries, but too much overlap creates near duplicate results and wastes the model's context window.
For each strategy, measure whether the correct source appears in the top results, whether the selected passage contains enough information to answer, and whether irrelevant material is introduced. Review examples by hand; a single aggregate score can hide high risk failures.
Use a retrieval pipeline
Keyword search is useful for exact IDs, error messages, and product terms. Semantic search helps when the query paraphrases a source. Hybrid retrieval can combine both, followed by a reranker when the added latency and cost are justified. Bound the number of passages and total context size before generation.
Apply user and tenant permissions before presenting retrieved content to the model. A reranker is not an authorization control. If access rules change, reindex metadata or block the affected source until the index is current.
Evaluate answers separately
If the correct passage is absent, fix ingestion or retrieval. If it is present but the answer is wrong, investigate the answer prompt, model behavior, or conflicting sources. Score groundedness and citation accuracy with human review for a sample of answers. A fluent answer with an unrelated citation is a failure.
Include "no answer" cases. The assistant should say the approved collection does not support an answer rather than fill a gap with a plausible guess. Track retrieval misses and user corrections after launch to improve the test set.
Common mistakes
- Changing chunk size without a before and after question set.
- Indexing navigation or repeated boilerplate as if it were useful content.
- Mixing obsolete and current policy versions.
- Relying only on answer fluency as an evaluation metric.
- Filtering confidential passages after retrieval or after generation.
Conclusion
Chunking succeeds when relevant, permission checked evidence survives ingestion and appears in the answer context. Measure retrieval and answer quality separately. Our AI automation service describes where this fits in an operational product.
References