The question that gets answered three different ways
"What's our remote-work policy?" Ask three people on a Tuesday and you get three answers: the version in the 2023 handbook PDF, the version someone's manager remembers, and the version that's actually in force after legal updated it in March. Knowledge management teams know this pattern cold — the same handful of policy questions arrive weekly, and the cost is not just time spent hunting. It's the quiet decisions people make on stale information because finding the current rule was harder than guessing.
That is exactly why policy question answering is the right first thing for a knowledge team to put AI behind — and exactly why it's dangerous to do casually. The win is real: employees stop interrupting HR and finance to ask where the expense cap lives, and they stop acting on a paragraph that was superseded two quarters ago. The risk is equally real: a confident answer pulled from an archived document becomes, in the employee's mind, the policy. The San Francisco Fed's research on AI and small businesses lands on the variable that decides which way it breaks — trust and the capacity to implement well. If people don't trust the answers, they route around the tool and you've shipped expensive shelfware. If they trust answers the system shouldn't be giving, you've automated a liability.
So the first build is narrow on purpose: answer the low-stakes, high-frequency questions with a citation to the live approved source and a visible effective date — and refuse to behave as if HR, legal, or finance handed their judgment to a bot.
The retrieval layer is the whole product
Most teams treat the model as the project. For policy Q&A, the model is the easy part. The product is the source hierarchy you build before the first answer goes out. When two documents disagree — and they always disagree — the system needs a deterministic rule for which one wins: current legal-approved handbook language outranks a wiki page, the HRIS or payroll system of record outranks a Slack thread, and a regional exception outranks the general rule for anyone in that region. Without that ranking, the model retrieves whatever scores highest on text similarity, which is frequently the most detailed old document, not the correct one.
This is where NIST's AI Risk Management Framework earns its place. Use it to draw the line between answer support — "the standard mileage reimbursement is X, effective this date, here's the source" — and judgment the tool must never simulate, like whether a specific leave request qualifies. The same instrument gives leadership the things to actually measure: answer acceptance, escalation quality, and whether governance is real or decorative.
Then there's the data the policy corpus touches. Handbooks reference compensation bands, leave entitlements, employee records, and region-specific obligations — so retrieval permissions and logging are not an afterthought. CISA's guidance on securing data used to train and operate AI systems should translate directly into who can retrieve what, and a log of every answer the system gave so you can trace a bad one to its source. Concretely, three behaviors are non-negotiable from day one: every answer cites its source and shows the effective date; low-confidence answers say so out loud instead of bluffing; and any question touching discipline, compensation, individual leave eligibility, or a regional legal nuance routes to the named owner rather than getting answered.
Pick one policy family, then watch what employees actually ask
Don't boil the handbook. Start with one policy family where the answers are stable and the stakes are low — travel and expense is a clean first choice: where the policy lives, what the current reimbursement limit is, which form starts the workflow, who approves it. Say a 120-person services firm starts here. Within a few weeks the query log tells you something no policy author knew: nobody can find the per-diem rule for international trips, and "what's the cap" gets asked dozens of times a month. That log is the real deliverable. It points at exactly which pages confuse people and which regional exceptions deserve a proper source instead of tribal memory.
Green light to expand when policy owners can approve the source set, agree on the boundary between answerable and escalate-only, and review the first batch of real queries. Hold when policies actively conflict, when nobody maintains effective dates, or when a leader is quietly hoping the tool will resolve exceptions for them — it won't, and asking it to is how you manufacture the liability. The scale decision rides on three signals: answers accepted with their citation intact, policy-owner corrections captured and fed back, and a visible drop in sensitive questions getting answered without a human in the loop. If those don't move, the problem is ownership, permissions, or source quality — fix that before adding more automation, not after.
Build it small: a citation-backed assistant over a curated corpus for the stable questions, and custom routing only where role, region, or approval status must change the answer. The maintenance loop is part of the product, not a phase-two nicety — the query log is a continuous diagnostic of where your policies are unclear. Once travel and expense is proving deflection and clean escalation, the same machinery extends to the next policy family, which is where it connects to broader manual-work triage and the larger AI transformation roadmap. Monday's move: open your help-desk or HR inbox, count which policy questions repeat, and pick the one family where the answer is stable and the source is already legal-approved. That's your pilot.
