AI knowledge bases: make answers traceable to the right documents

An answer can include a citation and still be wrong. The document may be outdated, written for another audience or contradicted by a more specific rule. Building an AI knowledge base therefore involves deciding which information is authoritative, how it is retrieved and how a reader can verify the resulting answer.

This guide uses a fictional professional membership organisation preparing an assistant for renewal questions. It has current guides, older forms, committee minutes and draft policy changes. The example is instructional, not a claim that Edeverse has deployed this system or that one architecture guarantees accuracy.

The method also applies to a personal research collection, an independent professional's service documents or an internal team library. Begin with the questions you want answered, then organise the evidence needed to answer them.

Start with questions rather than an archive

Importing every file creates a large collection, but not necessarily a useful knowledge base. The membership organisation should first list the questions members actually ask: which documents are required, when renewal closes and what happens when information is missing.

For each question, identify the approved source and the person responsible for its meaning. If nobody can name the authoritative answer, retrieval technology cannot settle the policy by itself.

Create a small coverage map:

Question	Approved evidence	Gap to resolve
Which renewal form should I use?	Current renewal guide and form register	Confirm which legacy forms are retired
What is the closing date?	Dated notice for the relevant renewal period	Avoid reusing last year's notice
Can I submit missing evidence later?	Approved procedure	Clarify any category-specific exception
Has my application been accepted?	Individual case status, not a general document	Requires a separate authorised lookup

The last question illustrates a useful boundary. A document explaining how acceptance works cannot establish the status of a particular person's application. Keep general knowledge and live operational records distinct.

Give every source a practical identity

A filename alone is rarely enough. Record a stable identifier, title, owner, approval state, effective period, intended audience, access level and location of the original.

Distinguish publication date from effective date. A policy published today may take effect next month. A recently edited document may only contain formatting changes. “Newest file” is not a reliable replacement for an explicit applicability rule.

For multilingual material, identify whether a translation is approved and which version it corresponds to. Two files with the same title in different languages may not contain the same current rule. If one translation lags behind, flag that rather than silently presenting it as equivalent.

Give superseded material a deliberate treatment. It may need to remain available for historical questions while being excluded from answers about the current renewal period. Do not delete useful history merely to simplify search, but do not mix it with current guidance without metadata.

A compact source register makes later updates easier. The owner can see which source needs review and which answers depend on it.

Resolve version conflicts before they reach the answer

Consider an original fictional example. An older renewal guide says supporting evidence may arrive within ten days. A current approved notice changes that to seven days for the next renewal cycle. Committee minutes discuss a possible five-day rule that was never adopted.

A query for “missing renewal evidence” might retrieve all three. The correct answer depends on approval state and renewal period, not which passage sounds most specific or appears first in search results.

Write the precedence rule with the policy owner. The current approved notice governs the relevant cycle; the old guide is historical; the committee discussion is not a rule. Store those distinctions where retrieval and answer generation can use them.

If two approved sources genuinely conflict and no precedence has been established, the assistant should identify the conflict and refer it to the owner. It should not average the numbers or choose the more convenient deadline.

Retain the conflict as a test case after it is resolved. Version problems tend to recur whenever a new notice, translation or exception is introduced.

Understand the retrieval and answer chain

Retrieval-augmented generation, usually called RAG, retrieves relevant material and supplies it to a model when generating an answer. Microsoft describes the pattern as retrieval, augmentation and generation, with indexes helping organise searchable content. This explains an architecture, not a guarantee that the correct passage will always be found or applied. Source: Microsoft Foundry RAG overview.

For the membership example, the practical chain is: interpret the question, identify the applicable audience and period, retrieve candidate passages, apply source precedence, produce the answer and attach checkable references.

Each stage can fail differently. Search may miss the relevant notice. A filter may select the wrong renewal year. The model may find the right passage but omit a condition. The citation may lead to a document the user cannot open.

Keep enough evidence to distinguish those faults. If the right passage was never retrieved, rewriting the final answer instruction may not help. If retrieval is correct but interpretation is wrong, importing more documents may make the problem harder.

Distinguish direct search, RAG and MCP

Direct search may be sufficient when the user wants a document or exact passage. There is no need to generate an explanatory answer for every lookup. For an individual maintaining a small collection, a clear folder structure and reliable search can be the simplest useful solution.

RAG becomes useful when an answer needs to combine or explain retrieved information. Its quality still depends on source preparation and retrieval. It is not a synonym for uploading files or retraining a model on your entire archive.

MCP concerns how an AI application accesses capabilities exposed by a server. A knowledge service might expose a search tool or document resource through that interface. The presence of MCP does not tell you whether the underlying search uses keywords, vectors, an index or a live document lookup. Source: MCP server capabilities.

Ask a supplier to draw the actual route. Where do documents come from? Are copies indexed? How are permissions applied? How does a change become visible? These questions reveal more than treating RAG and MCP as competing product features.

Clean the document parts that cause real errors

Inspect a sample of extracted text before indexing the full collection. A PDF that looks readable to a person may produce broken headings, repeated footers or missing table labels when processed.

For scanned material, check whether text extraction preserves dates, numbers and negation. A missed “not” can reverse a rule. An unreadable scan should be marked for repair rather than quietly treated as reliable evidence.

Tables deserve particular attention. In a renewal table, a value may only make sense with its row category, column heading and footnote. If extraction separates those elements, a correct number can be attached to the wrong membership class.

Remove duplicate boilerplate that overwhelms search without adding meaning. Preserve meaningful titles, section numbers and links to the original. Do not rewrite approved policy while cleaning formatting; substantive corrections belong with the source owner.

Choose a representative extraction sample: a short page, a long guide, a table, a scan and a document with footnotes. Fixing these patterns early is more efficient than diagnosing them only after a confident answer has gone wrong.

Divide documents without losing their meaning

Search systems often work with passages smaller than the complete document. The useful size depends on structure and the questions being asked. Microsoft's chunking guidance discusses fixed, structural and semantic approaches, including preserving context across document sections. Source: Azure AI Search document chunking.

For our example, a passage saying “submit within seven days” needs the surrounding conditions: seven days after what, for which renewal period and for which member category? Keep those qualifiers attached or available through metadata.

A practical review procedure is to inspect retrieved passages without their full documents. Can a reviewer tell what rule each passage states and when it applies? If not, the chunks may be too small or missing useful headings.

Larger passages are not always better. A whole handbook can bury the relevant sentence among unrelated exceptions. Compare several plausible configurations using your actual questions rather than adopting one chunk size as a universal answer.

Preserve a path back to the full context. The final answer may only need a short passage, but the reviewer should be able to inspect the source section if a condition is unclear.

Test retrieval before judging the generated prose

For each question in the coverage map, record the expected source or passage. Run the retrieval stage and inspect what it returns before asking the model to write an answer.

Check whether the right source appears, whether obsolete versions crowd it out and whether the filters respect audience and effective period. Try different wording for the same intent. Members may ask “renewal deadline”, “last day to renew” or “how long have I got?”

Include exact identifiers too. A form number or policy code may be easier to retrieve with an exact match than a broad semantic similarity. The appropriate search method depends on the material; compare approaches with the examples rather than assuming a single method is always superior.

Separate retrieval failures from legitimate absence. If no approved source answers a question, the correct outcome is a gap for the content owner, not an invitation to invent a source.

Only after retrieval works should you assess the full answer. Check that the model includes the important conditions and does not add a claim that the retrieved material does not support.

Make citations useful for verification

A citation should support the nearby factual claim. A link to the organisation's homepage is weak evidence for a specific deadline. Prefer the relevant document, section or stable passage reference where the system supports it.

For an answer with several independent claims, show which source supports each. A single citation at the end of a long paragraph can make it unclear whether all the statements are grounded.

Open the citation as the intended user. The assistant may have access that the user does not. If the answer exposes information from a restricted source, a working internal citation does not make the disclosure appropriate.

Test a deceptively plausible answer with an irrelevant citation. Reviewers should assess whether the source actually entails the claim, not merely whether a link exists. This is especially important for dates, eligibility conditions and exceptions.

Where the source conflicts or remains incomplete, say so. A traceable uncertainty is more useful than a polished conclusion that cannot be checked.

Verify permissions with more than one user profile

A knowledge assistant used by several people must apply the relevant access boundaries for each person. The fact that the indexing account can read a document does not mean every user of the assistant should receive its contents.

Create harmless sample documents with distinct markers. Give one test user access to the general renewal guide and another access to an additional internal procedure. Ask equivalent questions from both profiles and inspect the retrieved material and answers.

Check indirect leakage too. A summary, title or quoted fragment can reveal restricted content even if the final document link is blocked. Permission testing should inspect the result, not only whether the user can open the original.

For a personal collection, decide which parts may be used for each task. A home-project question does not require unrelated financial records simply because both are stored in the same account.

If the system cannot enforce the necessary distinction, reduce the source set or user scope for the pilot rather than relying on a general instruction to be careful.

Prove that updates reach the final answer

A document update is not complete until the assistant uses the new applicable information. The source may change while an index, cached result or copied passage remains old.

Use a harmless controlled update test. Start with an approved sample rule, ask the relevant question and record the answer. Change the sample rule through the normal contribution process, wait for the expected update path and ask again.

Inspect each stage: source version, extracted content, index or retrieval result and final answer. If the answer remains old, this trace shows where the update stopped.

Test withdrawal too. When a source is retired, determine whether it should disappear from current answers, remain available for historical queries or be removed entirely. Those are different requirements.

Document the expected delay and the owner of a failed refresh. “The folder is connected” does not tell you whether changes arrive immediately, periodically or only after a manual action. The operating procedure should make that clear.

Organise contribution without losing source authority

Make it easy for people to report a missing answer, but separate a suggestion from an approved rule. A useful contribution route captures the question, the proposed source change and the owner who can approve it.

Avoid letting a conversational correction silently rewrite policy. A user saying “I think the deadline is Friday” may be useful evidence of confusion, not authority to change the renewal guide.

Review repeated unanswered questions. They may indicate a retrieval weakness, but they may also reveal that the organisation never documented an important procedure. Improve the source where appropriate instead of adding more instructions to compensate for missing knowledge.

Keep the source register current when owners change. An archive with no responsible editor gradually becomes difficult to trust, regardless of the search technology.

Prepare a small reusable acceptance set

Before launch, test a current factual question, an old-period question, a source conflict, a missing answer, a restricted document and an updated rule. Include a table-based answer and a question phrased differently from the source heading.

For each, retain the expected source, the important conditions and the acceptable response when information is insufficient. Reuse these cases after changes to extraction, search, permissions or instructions.

Measure correct supported answers and appropriate acknowledgement of missing information separately. Count inaccessible citations, obsolete references and unsupported additions. The number of imported files is not a quality measure.

Expand the corpus when it adds coverage for real questions. A smaller collection with clear authority and working updates can be more useful than a vast archive whose contents the assistant cannot distinguish.

Frequently asked questions

Should I upload the entire archive first?

Usually it is more useful to begin with the questions and approved sources for a bounded task. Add documents when they fill a real gap. Separate historical and draft material so it does not compete with current guidance without context.

Does a citation prove that an answer is correct?

No. The cited passage must support the specific claim, apply to the relevant date and audience, and be appropriate for the user to access. Test those points rather than counting links.

What should happen when two approved sources disagree?

Apply an established precedence rule if one exists. Otherwise identify the conflict and refer it to the responsible owner. The assistant should not invent a resolution or choose a version merely because it was retrieved first.