The 2023 refund policy is still answering questions
Here is the failure mode nobody demos. A support rep asks the new internal search tool, "What's our refund window for annual plans?" The tool answers instantly, confidently, in a clean paragraph. The answer is wrong — it pulled from a policy doc that was superseded eighteen months and one pricing change ago. Nobody catches it, because the answer looked exactly as authoritative as a correct one. That is the whole problem with internal knowledge search: a confident wrong answer and a confident right answer are visually identical.
So the question that actually matters on day one is not "can the AI find documents?" Of course it can. The question is "when it finds the wrong document, does anyone know?" A search layer that always produces a fluent paragraph has hidden the one signal you needed — that the underlying source was stale, contradicted by a newer file, or sitting behind a permission wall the asker shouldn't cross. The U.S. Census Bureau's data on AI use at businesses shows adoption climbing fast; the OECD's research on AI adoption by SMEs shows smaller firms adopting under real resource pressure. Neither tells you the answers are right. That gap is where the work is.
Reframe the build. You are not deploying a chatbot over the company drive. You are deciding which questions an AI is allowed to answer from documents — and forcing it to cite which document and how fresh it is, every single time. Pick one bounded corpus to start: say, the support team's policy and SOP set, not "all of Confluence plus the shared drive plus four Slack channels." One queue of questions, one library, one named owner who can vouch for what's in it.
Three numbers that tell you if retrieval is honest
Before you let internal search touch a customer-facing or money-touching workflow, you need to see its work. Make every answer ship with four things attached: the exact source document it pulled from, the document's last-reviewed date, the asker's permission scope, and — critically — a "no confident source" flag when the corpus can't actually answer. That last one is the feature most tools omit and the one that separates a useful system from a liability. The NIST AI Risk Management Framework makes the point that risk is contextual: the same retrieved sentence is harmless in someone's draft email and material the moment it becomes the official answer a rep reads to a customer.
Then watch three numbers during the first weeks, because they fail in distinct ways. Citation rate — what fraction of answers link to a real, current source versus paraphrasing from nowhere. If this is low, your retrieval is hallucinating connective tissue. Stale-source suppression — how often the system declines to answer from a document past its review date. If this is near zero, you're shipping expired policy with a straight face. Permission-denial accuracy — whether the system actually withholds the comp doc from the person who shouldn't see it. The CISA AI data security best practices are blunt about this: the boundary lives at the data layer, not in a system prompt that asks the model nicely to behave. And the enterprise privacy commitments from your model vendor govern what leaves your walls, but they don't govern who inside your walls is allowed to ask what.
When the same wrong answer keeps surfacing, resist the urge to tune the model. The Deloitte State of AI in the Enterprise 2026 finding that only about a quarter of leaders report genuinely transformative impact maps cleanly onto this: the repeated error is almost never a model problem. It's two conflicting SOPs both sitting in the corpus, or a doc nobody dared delete. Fix the library, not the algorithm.
A 30-60-90 you can actually run Monday
Days 1-30: don't connect anything to AI yet. Take your one starting corpus — say a 40-person company's support policy and SOP folder — and have the owner walk it document by document. For each file, answer one question: would you defend this answer if a customer quoted it back to you? Anything that gets a "no," or a "well, it depends," gets archived out of the searchable set or tagged with a review date. You will be surprised how much of the drive your own owner won't stand behind. That cleanup is the project; the AI is the easy part.
Days 31-60: turn on retrieval over the cleaned set and run it in shadow mode. Every answer the system produces, a trained rep also answers the old way. Compare. You're not measuring whether the AI is fast — it's always fast. You're measuring whether it agrees with the person who actually knows, and whether its citations point where they should. Log every disagreement; each one is either a corpus problem or a retrieval problem, and the citation trail tells you which.
Days 61-90: make the scale-or-stop call. A good outcome is boring — reps stop pinging the #ops channel for policy answers, the "no confident source" flag fires honestly when the library has a gap, and exceptions go down. A bad outcome looks slicker and quietly puts managers back to checking answers by hand, which is just the old work plus a new tool to babysit. If you're weighing this against other first AI projects, the AI Opportunity Score helps you sequence it; once shadow mode produces real evidence, the AI ROI Calculator tells you what the time saved is worth. We sequence that whole path inside the AI Transformation Blueprint so internal search becomes the first governed workflow, not a confident wrong-answer machine.
