Pull five documents and read them out loud
Here is the test I run before anyone spends a dollar on a knowledge bot. Pick the five questions your team actually asks most — the refund window, the warranty exception, the approval threshold for a discount, the onboarding step everyone forgets, the policy nobody can find. Now go pull the documents that answer them. Read each one out loud to the person who owns that area.
What happens next tells you everything. Half the time the document is two pricing changes out of date. Sometimes there are three versions of it living in three different folders, and nobody can say which one is current. Occasionally the "document" is just a Slack message from someone who left last spring. A knowledge bot does not fix any of that. It retrieves whatever you point it at and wraps it in a confident sentence. If the source is stale, you have built a faster way to be wrong, and you have given it a chat box so more people trust it.
Retrieval-augmented generation — RAG — earns its keep only when the answer should come from documents you have vetted rather than the model's guesswork. That is a real and useful constraint for an SMB drowning in repeated questions. But the San Francisco Fed's small-business AI analysis shows interest climbing far faster than readiness. A knowledge bot is not an AI project with a documentation step bolted on. It is a documentation project that happens to end in a chat interface.
The four places a knowledge bot quietly breaks
If your five documents survived being read aloud, good. Now stress the four joints where these projects actually fail — none of which show up in the vendor demo.
The first joint is permissions. Your HR comp bands, the founder's email thread about a layoff, the customer contract with the non-standard SLA — those sit in the same drives as the support FAQ. A bot that ingests "the knowledge base" will happily answer a front-line rep's question with executive compensation data unless someone designs what each role is allowed to retrieve. For most SMBs this is the hardest part, because access was never deliberate to begin with.
The second is retrieval accuracy. Take ten real questions, write down the document that should answer each, then watch what the system actually pulls. A bot that's right eight times out of ten sounds great until you remember the other two answers arrive with the exact same confident tone. Your team can't see which is which.
The third is ownership of bad answers. When the bot says something wrong, who finds out, and who fixes the source? If the answer is "nobody," you have built a complaint generator. The OECD's report on AI adoption by SMEs keeps landing on the same point: the binding constraint isn't appetite, it's the operating capacity to maintain what you launch.
The fourth is maintenance economics. If your source documents change weekly and no one owns the update cadence, answer quality decays on a predictable curve. A failed test here is not a verdict that RAG is wrong for you. It's a signal that your first project is document cleanup, access design, or naming an owner — not the bot.
Start with one team and a logbook, not a launch
When the documents and the four joints hold up, build narrow. Pick one team — support is usually the cleanest, because its questions repeat and its sources are concrete. Give the bot only that team's vetted documents, not the whole drive. Then keep a logbook from day one: every question asked, every answer that was wrong, every time the system retrieved nothing because the document simply didn't exist.
That logbook is the entire point. Within a few weeks it stops being an AI metric and becomes a map of your knowledge debt — which sources are missing, which owners are behind, which questions you didn't know your customers were asking. The Deloitte State of AI report ties value not to the model but to the operating change around it. A knowledge bot delivers that change only when it forces your library to get better. Bolt it on top of files employees already distrust and you've automated the distrust.
Monday's move: run the five-document test on your own team. If three of the five come back stale, contradictory, or missing, you've found your real project — and it isn't AI yet. When retrieval genuinely is the right answer and you want a build that comes with source control and a review cadence rather than just a chat box, our work on AI Knowledge Systems and RAG is where to look next. Use Evaluate a RAG build when the repeated questions are costing you real time but you want the source model proven before you point a bot at it.
