AI Knowledge Systems Explained

Every established business has the same quiet problem. The knowledge that makes it work is written down somewhere, but nobody can find it quickly. Policies sit in a shared drive. Product details are spread across old proposals and a wiki nobody has updated since the last reorganisation. The definitive answer to a common customer question exists in an email thread from two years ago.

New staff take months to become useful, not because the work is hard, but because knowing where to look takes months. Experienced staff spend a surprising share of their week answering questions they have answered before.

Retrieval-augmented generation, usually shortened to RAG, is the current best approach to this problem. It is worth understanding what it does and does not do, because it is easy to expect the wrong thing from it.

What RAG actually is

A general-purpose language model knows a lot about the world and nothing about your business. Ask it about your refund policy and it will produce something plausible and wrong.

RAG closes that gap with a straightforward arrangement. When someone asks a question, the system first searches your own documents for the passages most likely to contain the answer. It then passes those passages to the model along with the question and an instruction: answer using this material, and say so if it is not there.

The model supplies the language skills. Your documents supply the facts. The answer comes back with references to the source, so the reader can check it.

The mechanics behind the search step involve splitting documents into sections, converting them into numerical representations that capture meaning, and matching them against the question. The practical consequence of this is what matters: the system can find a relevant passage even when the question uses none of the same words as the document. Someone asking “how long do I have to send something back” can reach a policy written entirely in the language of “returns window”.

What it is good for

RAG suits situations where the answer exists in writing but is hard to locate.

• Internal support. An assistant that answers staff questions about policies, procedures, and systems, citing the actual document.
• Customer service assistance. Giving agents a fast, sourced answer instead of asking them to search a knowledge base mid-call.
• Onboarding. Letting new starters ask the naive questions they would otherwise hesitate to ask a colleague.
• Technical and product reference. Searching specifications, manuals, and configuration guides that are too large to read.
• Contract and document review. Finding the clauses relevant to a specific question across a large set of agreements.

The common thread is that a person currently spends time searching, and the cost of that searching is significant across the organisation.

What it will not do

Being clear about the limits saves disappointment.

It cannot answer what is not written down. If the real policy lives in a manager’s head, RAG will not find it. Building one of these systems often exposes how much of your knowledge is undocumented, which is uncomfortable but useful.

It inherits your document quality. Contradictory, outdated, or duplicated documents produce contradictory, outdated answers, presented with the same fluent confidence as correct ones. Where two versions of a policy exist, the system has no way to know which one you meant.

It is not a database. RAG answers questions about text. Questions like “how many orders shipped late last month” belong to your reporting tools, not a document retrieval system. Some setups combine both, but the distinction should be deliberate.

It is not automatically accurate. Grounding the model in real sources reduces invention considerably. It does not eliminate it. Citations, so that a person can verify, remain important.

What building one actually involves

The model is rarely the hard part. The work concentrates in these areas:

Stage	What it involves
Source selection	Deciding which documents count as authoritative
Preparation	Extracting clean text from PDFs, wikis, and file shares
Chunking	Splitting documents so retrieved passages are complete and coherent
Permissions	Ensuring people only retrieve what they are allowed to see
Evaluation	A test set of real questions with known good answers
Maintenance	Keeping the index current as documents change

Two of these deserve emphasis. Permissions must be enforced at retrieval, not by asking the model to be discreet. If HR documents are in the same index as general policies, the retrieval step itself needs to filter by the user’s access rights.

Evaluation is what separates a system people trust from one they quietly stop using. Before launch, collect thirty or forty real questions with agreed correct answers, and measure against them. Repeat that measurement after any change. Without it, you are guessing.

A sensible first project

Start with one well-bounded body of knowledge and one clear audience. A single department’s procedures, or the product documentation for one line. A narrow scope makes the document quality manageable and the evaluation honest.

Before building anything, do a short inventory: what documents exist, which are current, who owns them, and where the contradictions are. That inventory has value on its own, and it will tell you whether a retrieval system is worth building yet.

Then treat the first version as a tool for a small group rather than a company-wide launch. Watch which questions it handles well, which it misses, and which reveal gaps in the underlying documents. The failures are the most useful output of the first month.

If you are considering something along these lines, we are happy to talk through what a retrieval system would involve for your particular content, including the parts that tend to be more work than expected. It is often worth a conversation before committing to a build.