Test the evidence behind the answer
An internal AI assistant should be evaluated on whether it finds the right source material for a task. Begin with questions your team actually needs answered: which product version a procedure covers, which document defines a requirement, or where an exception is explained. For each question, have a knowledgeable reviewer identify the supporting passages. Include questions that the available documents cannot answer. This gives the team a practical reference set before changing the search system.
Compare exact terms and meaning
Microsoft documents hybrid search in Azure AI Search as combining full-text and vector queries, then merging their results. Keyword search can be useful for product codes and exact terminology; vector search can find conceptually similar material. Our recommendation is to compare keyword, vector and hybrid retrieval on the same reference questions. Keep the documents and evaluation criteria fixed so a configuration change can be assessed fairly. Microsoft: Hybrid search overview.
Keep document context with each passage
Anthropic’s Contextual Retrieval method adds a short explanation of a passage’s place in its document before indexing it for retrieval. Its published experiments provide a reason to test the approach, rather than a guarantee for another organisation’s data. In your own trial, inspect whether retrieved passages retain the title, version and other context needed to interpret them. Check generated context against the source before relying on it. Anthropic: Contextual Retrieval.
Measure retrieval and answers separately
Our suggested review records whether the expected evidence appears among the retrieved passages, whether the answer accurately uses it, and whether the displayed citation leads to that evidence. Test restricted material with users who should and should not have access. Keep access checks in the retrieval system; an instruction to the language model is insufficient as an access control. When evidence is missing, the assistant should explain the gap and offer a human review path.
Choose the improvement your workflow needs
Compare useful answers, retrieval time and operating cost on the tested workload. If you add a reranking stage to reorder candidate passages, measure its effect on the complete request. Keep examples of failures, assign their causes, and repeat the reference questions after changing documents or retrieval settings. Start with one workflow whose subject experts can review the results. Expand when the evidence supports the next step.