Start with the question the assistant must answer A document chatbot is useful when people need answers grounded in approved knowledge, such as product doc...
Start with the question the assistant must answer
A document chatbot is useful when people need answers grounded in approved knowledge, such as product documentation, policies, or internal procedures. Retrieval augmented generation (RAG) searches that knowledge and sends relevant passages to a language model. The model can then answer with source references. This does not guarantee correctness: retrieval can miss the right passage, the source may be stale, or the model may infer more than the evidence supports.
Before selecting a vector database, collect representative user questions. Include questions that should be answered, questions that need a clarification, and questions the assistant must refuse because the user lacks access or the documents do not contain the answer.
Build an ingest pipeline with provenance
For every source, retain document ID, version, title, update time, owner, and access group. Extract text, remove navigation noise, split the content into coherent passages, and keep the passage's location in the original document. A citation is only useful if it leads a reviewer back to the exact source.
Chunk size is a tuning choice. Tiny chunks lose context; very large chunks reduce retrieval precision and increase model input cost. Test several chunking strategies against your actual question set. Reprocess changed documents and remove superseded passages so the index does not mix current and obsolete instructions.
Apply permissions before retrieval results reach the model
An internal assistant must not retrieve a confidential passage and hope the prompt tells the model to ignore it. Enforce document access in the retrieval query using the authenticated user's organization and permissions. Verify the filter on both keyword and vector search paths. If a document's permission changes, update the index metadata promptly and consider a fallback that blocks access until the change is complete.
Keep audit records for retrieval decisions without logging entire sensitive prompts or documents unnecessarily. Use the same tenant boundary for chat history and uploaded files.
Retrieve, rerank, and answer
A practical pipeline is: normalize the question, retrieve candidate passages, rerank them, select a bounded context, then generate an answer with citations. Hybrid keyword and semantic search can help when exact identifiers and natural language appear in the same collection. Reranking may improve relevance, but it adds latency and cost; measure the tradeoff.
Tell the assistant to distinguish source statements from its own inference. If no relevant passage is found, return a useful "I could not find that in the approved documents" response and suggest the next human contact or search path. Do not manufacture a citation.
Evaluate with a representative set
Create a small evaluation set from real questions with approved reference answers and source passages. Score retrieval separately from answer generation. If the correct passage was never retrieved, changing the answer prompt will not fix the root problem. Review groundedness, citation accuracy, refusal behavior, latency, and cost.
Include adversarial questions: a user asks about another tenant, a document contains instructions that try to redirect the model, or an obsolete policy contradicts a current one. Run these tests after changing chunking, embedding models, access filters, or prompts.
Operate the system after launch
Monitor unanswered questions, retrieval misses, permission denials, source freshness, provider errors, latency, and usage cost. Provide a feedback path that lets a reviewer correct a source or flag a misleading answer. Version the prompt and index configuration so regressions can be traced to a specific change.
For high impact decisions, keep a human approval step. A document assistant should help users find information; it should not autonomously change account rights, approve payments, or make legal or medical determinations merely because it produced a confident sentence.
Common mistakes
- Using vector similarity without metadata access filtering.
- Treating a citation as proof when it points to an unrelated passage.
- Indexing a source once and never removing stale versions.
- Measuring answer fluency while ignoring retrieval recall.
- Logging full confidential documents for debugging.
Conclusion
The useful unit of a RAG system is a permission checked, versioned source passage tied to an evaluated answer. Build the question set and access model first; choose infrastructure after those requirements are clear. Our AI and business automation service describes how this fits into an operational application.
References