
RAG Knowledge Base: How It Works and How to Build One
A RAG knowledge base supplies external information to an AI application so it can answer questions using relevant source material. RAG stands for retrieval-augmented generation: the application retrieves evidence, adds it to the model’s input, and asks the model to produce an answer.
For a business, that evidence might come from product documentation, support articles, policies, or technical runbooks. The useful outcome is an answer that fits the question, reflects the applicable information, and points readers to supporting sources.
Building that capability takes more than uploading files. You need to decide which information is authoritative, who may retrieve it, how changes become searchable, and how you will detect an answer that sounds convincing but is wrong. This guide walks through those decisions and a practical way to launch a bounded pilot.
On this page: How RAG works · When to use it · Build the knowledge base · Protect access · Evaluate results · Choose a platform.
What Is a RAG Knowledge Base?
The term can refer to the collection of documents an application searches or to a service that manages ingestion and retrieval. Keep three parts separate when planning your system:
- The source collection: the maintained articles, files, records, and other approved information. This is where owners correct policies and document product changes.
- The retrieval layer: the search indexes, connectors, and query logic that locate useful evidence. An index is usually a derived representation, so changing the source does not necessarily change search results immediately.
- The answering application: the user interface, identity checks, context assembly, language model, citation display, and failure handling.
An ordinary searchable knowledge base returns articles for people to read. A RAG application can use retrieved passages to compose a response or explain information across sources. The original articles remain important: they provide context, ownership, and a destination for readers who need to verify an answer.
Retrieving documents does not, by itself, train the generation model on those documents. The application supplies selected information when a question arrives. Microsoft’s RAG and indexes documentation describes this separation between retrieval, added context, and generation.
RAG can reduce unsupported answers, often called hallucinations, but it cannot guarantee correctness. IBM’s RAG overview explicitly notes that the approach does not make a model error-proof. Relevant retrieval, accurate sources, and careful answer evaluation still matter.
How a RAG Knowledge Base Works
A typical implementation has a preparation workflow and an answering workflow. Keeping them separate makes failures easier to diagnose: missing source content is a different problem from a model misreading good evidence.
Prepare the information for retrieval
In a common vector-based pipeline, a connector reads approved sources, a parser extracts usable content, and a chunking process divides longer documents into passages. An embedding model converts those passages into numerical representations. The index stores those representations with the text and metadata needed to identify the source. AWS illustrates this process in its knowledge-base preprocessing and runtime diagrams.
Retrieve evidence and generate the answer
- Receive the question and verified user context. Establish the audience and applicable access rules.
- Find eligible evidence. Search authorized content using a suitable retrieval method.
- Select the useful passages. Remove irrelevant repetition and optionally rerank candidates.
- Assemble the model input. Include the question, selected evidence, source identifiers, and answer instructions.
- Return an answer or a clear limitation. Show supporting citations, request clarification, or explain that the available evidence is insufficient.

A vector database is common, but it is not a requirement for every RAG system. Keyword search, structured queries, or combinations of retrieval methods can supply evidence. For example, AWS documents retrieval from structured data stores using SQL without converting that data into vector embeddings. Choose the retrieval method for the information you need to find.
When to Use RAG—and When Another Approach Fits
Start with the task, then choose the architecture. These approaches can work together; the table identifies what each contributes.
| Reader or business need | Approach to consider | Decision to make |
|---|---|---|
| Answer questions across changing company documents | RAG over approved, maintained sources | Can you retrieve the applicable evidence and keep it current? |
| Find an article, form, or exact reference | Conventional knowledge-base search | Would a clear result list solve the task without generated prose? |
| Analyze a small, known set of documents once | Supply the documents directly in the model context | Do they fit the model’s limits, access rules, and cost budget? |
| Improve a model’s recurring style or task behavior | Prompt design and, where justified, fine-tuning | Is the problem behavior, missing evidence, or both? |
| Answer a live account question or perform an action | An authorized database/API integration, optionally combined with RAG | Which system holds the current record, and who may read or change it? |
Microsoft distinguishes retrieval for private or changing information from fine-tuning for behavior or task performance. They can be combined. A retriever may find the correct policy while a tuned model follows a specialized response format.
For a first project, write a scope statement such as: “Help authenticated customers troubleshoot the current product using approved support articles.” That is easier to evaluate than “answer anything about the company.” If your source documents disagree about the answer, resolve that disagreement before expanding the assistant’s scope.
How to Build a RAG Knowledge Base
1. Choose a bounded use case and collect real questions
Identify one audience, one information domain, and the kinds of answers the assistant should provide. Gather representative questions from support conversations, search logs, or interviews, using information you are permitted to process. Include unfamiliar wording and questions your sources cannot answer.
For each question, record the expected source, the conditions that change the answer, and what a useful response must include. Have a subject-matter owner review these expectations. This becomes both a content backlog and the foundation of your evaluation set.
For example, a setup assistant might need to distinguish installation instructions by operating system and product version. An acceptable answer must match both. A generic installation article may be relevant to the topic while still being unsuitable evidence for that user.
2. Audit the sources before connecting them
Create a source inventory with an owner for every collection. Decide which sources are approved for answers, which are historical references, and which should remain outside the system. Exclude secrets, uncontrolled drafts, and material the intended audience must not receive.
Inspect the extracted output, not just the original file. A PDF may look correct while its extracted table loses column labels. A scanned troubleshooting guide may require OCR. Preserve prerequisites, exceptions, units, warnings, and the relationship between a table cell and its row and column headings.
Resolve duplicate and conflicting policies deliberately. A newer file timestamp does not necessarily mean a policy is effective now. Contentful’s RAG tutorial demonstrates why conflicting source material needs attention during evaluation; content governance remains necessary even when the pipeline runs successfully.
3. Preserve metadata that explains where an answer came from
Use a metadata contract across ingestion, retrieval, and citation display. The following is a practical starting design, not a required schema for every platform.
| Metadata | Practical purpose |
|---|---|
| Stable source ID and chunk ID | Connect a result to its parent source and replace or remove derived records reliably. |
| Title, canonical URL, and section or page | Give the reader a meaningful citation that opens the supporting material. |
| Product, version, language, and intended audience | Select evidence that matches the user’s situation. |
| Effective date, review date, and content status | Distinguish current, scheduled, historical, and withdrawn information. |
| Content version or hash and indexed timestamp | Identify what was processed and investigate synchronization lag. |
| Owner and source authority | Route corrections and resolve conflicts between sources. |
| Access-control and tenant metadata | Support enforcement of the authenticated user’s permissions. |
A metadata field does not enforce a rule on its own. Your application and retrieval service must actually use it. Test whether a version filter excludes obsolete instructions and whether a restricted source remains unavailable to an unauthorized user.
4. Split documents into chunks that retain meaning
A chunk should contain enough context to support the question it is likely to answer. Splitting at headings or logical sections is a useful candidate for structured documentation. Keep a procedure’s prerequisites close to its steps, and preserve table labels when extracting individual rows.
Microsoft documents fixed-size, structure-based, semantic, and combined chunking approaches. There is no universal best chunk size or overlap. Small passages can omit necessary qualifications; large passages can bring irrelevant material into the answer context. Overlap can preserve continuity but also duplicates content.

Test a few strategies against the same questions. Inspect the retrieved text before judging the final answer. If a passage says “restart the service” but omits the preceding warning about active jobs, change the chunk boundary or return the surrounding section. Enlarging the model will not restore evidence that retrieval never supplied.
5. Choose retrieval methods for the questions people ask
- Keyword retrieval is useful to test for exact names, error codes, part numbers, and quoted phrases.
- Vector retrieval compares numerical representations of meaning and can help find passages when the question uses different wording.
- Hybrid retrieval combines text and vector results. Evaluate whether it improves coverage across your exact-match and paraphrased questions.
- Reranking reassesses an initial candidate set before the application selects evidence for the answer.
These are distinct operations. Microsoft’s retrieval guide describes search choices and ranking strategies. Its semantic ranker documentation describes a later ranking stage over existing results. Product labels such as “semantic search” can refer to different mechanisms, so check what a service actually does.
Use a baseline search configuration, inspect misses, and change one meaningful variable at a time. If paraphrases fail, test vector retrieval. If exact identifiers disappear among similar passages, test keyword or hybrid retrieval. If the right evidence is present but buried, test reranking. Compare relevance, latency, and operating cost before keeping the change.
Set the retrieved candidate count and final context size through evaluation. A similarity score is not a calibrated probability that an answer is correct. Avoid borrowing a tutorial’s cutoff as a universal confidence threshold.
6. Define how the assistant uses evidence
Give each selected passage a source identifier, separate evidence from instructions, and define the expected answer format. Budget the model’s context for instructions, the question, relevant conversation history, retrieved material, and output. Deduplicate overlapping passages before using scarce context space. These are central concerns in Microsoft’s RAG prompt-engineering guidance.
Define the following behaviors as application requirements, then test them:
- Support material factual claims with citations to the passages that actually justify them.
- Ask a focused clarifying question when the applicable product, version, or situation is unknown.
- Explain when the available sources do not contain enough evidence.
- Identify unresolved source conflicts rather than silently merging incompatible instructions.
- Route live account data and actions to authorized integrations; route unresolved support cases to an appropriate human channel.
A useful no-answer response might say, “The available documentation does not specify that limit. I can show the published limits or help you contact support.” This is an illustrative response, not a verified product answer. A visible citation makes checking easier, but you still need to verify that it supports the claim.

A Practical RAG Knowledge Base Example
Consider a hypothetical customer-support assistant for a file-sync product. A customer asks: “Why does error SYNC-17 appear after upgrading to version 3?” The approved collection contains a version 3 troubleshooting article, an older version 2 guide, and an internal incident report.
- The application establishes the customer’s identity and searches content that customer is allowed to access.
- The retriever uses the error code and version information to find the applicable troubleshooting passage. The internal incident report is excluded from the customer’s context.
- The selected passage includes the documented cause, prerequisites, and recovery steps. The older guide is not used as though it describes the current version.
- The answer cites the relevant version 3 section. If the question asks about an undocumented account-specific condition, the assistant explains the gap and routes it appropriately.
This example creates concrete checks for a pilot: does retrieval preserve the error code, choose the applicable version, respect permissions, retain the warning, and cite the right section? A fluent response alone would not answer any of those questions.
Here is a miniature source-and-answer pair for that fictional product. S1 is an illustrative source label, not a real product citation.
| Illustrative source S1: version 3 troubleshooting | Illustrative answer supported by S1 |
|---|---|
| “After a version 3 upgrade, SYNC-17 means the saved connection token has expired. Pause active jobs, then reconnect the desktop app from Connection settings.” | “The version 3 guide attributes SYNC-17 after an upgrade to an expired connection token. Pause active jobs, then reconnect the desktop app from Connection settings. [S1]” |
The response preserves the prerequisite and ties the diagnosis to the applicable version. It does not invent a repair procedure or expose information from the internal incident report.
Protect Permissions and Treat Retrieved Content as Untrusted
Authenticate users and enforce document access
Permission to ingest documents, permission to call a search service, and an end user’s right to read a document are separate concerns. Microsoft’s query-time access-control documentation explicitly distinguishes the service caller from the delegated user whose document permissions are evaluated.

Preserve access metadata through parsing and chunking, and enforce access before unauthorized content reaches the model or user. Check citation previews and cached responses as well as retrieved passages. OWASP’s RAG Security Cheat Sheet covers these controls across the pipeline.
Derive tenant scope from authenticated context and enforce it on every retrieval and document operation. Do not trust an unverified tenant ID supplied by the client. Microsoft’s multitenancy guidance explains the application’s responsibilities across shared-index and separate-index designs.
Managed features still have an identity contract. AWS’s ACL-awareness documentation states that Bedrock Managed Knowledge Base relies on the application to authenticate users and supply verified user context. Connector support and permission freshness also vary. Ask a provider to demonstrate your actual source and identity setup, including what happens after access is revoked.
Keep retrieved instructions from gaining authority
A retrieved document can contain text that tries to change the assistant’s instructions or make it reveal information. OWASP identifies this as an indirect prompt-injection risk and notes that RAG does not eliminate it.
Treat passages as evidence. Keep tool permissions and sensitive actions under application controls, use the least privileges the workflow needs, and test malicious instructions embedded in otherwise relevant documents. Prompt wording and content screening can help, but they do not replace authorization or guarantee safe behavior.
For a documented-answer workflow, an inability to evaluate the required permissions or an unavailable retrieval service should produce a controlled failure. Do not silently substitute a confident answer from model memory when the required evidence could not be retrieved.
Keep the Knowledge Base Current
Freshness depends on the path from source change to usable evidence. Track when the source changed, when ingestion processed it, and when the new result became retrievable. A successful connection or completed job is not sufficient proof that every affected answer now uses the new information.
For example, AWS’s documented synchronization workflow processes added, changed, and deleted documents. Its resync table distinguishes these cases, and the documentation notes that visibility can lag synchronization completion.
Use different operational checks for different events:
- Content correction: retrieve the revised passage and verify that the superseded version is no longer presented as current.
- Source deletion: verify removal from the applicable indexes and derived chunks, then invalidate affected answer caches.
- Access revocation: test as the affected user and confirm that retrieval, citation previews, and cached answers respect the changed rights.
- Product or policy release: rerun the questions whose expected answers changed and check that old-version questions still receive appropriately labeled historical guidance when allowed.
Deleting a source for everyone and revoking one person’s access are different events. Maintain an owner and a propagation procedure for each. Keep an approved recovery path for ingestion failures, and investigate orphaned chunks or stale caches rather than assuming the original file controls every copy.
How to Evaluate a RAG Knowledge Base
Evaluate the retrieval stage and the generated answer separately. Microsoft’s RAG evaluator documentation distinguishes process evaluation from final-response evaluation. This lets you determine whether a failure began with missing evidence or with the model’s use of evidence.

Build a test set that includes failure cases
For every test, record the user context, question, expected evidence, required answer facts, and acceptable failure behavior. The following matrix is a suggested design for your own evaluation, not a report of measured results.
| Test case | What to verify |
|---|---|
| Answerable question | The applicable source is retrieved and the answer includes the necessary conditions and steps. |
| Paraphrase or exact identifier | Different wording works without losing meaningful codes, names, or version numbers. |
| Missing or ambiguous information | The assistant acknowledges the gap or asks a useful clarifying question. |
| Conflicting or outdated sources | The answer follows the approved authority/version rules or exposes the unresolved conflict. |
| Restricted content or another tenant | Unauthorized evidence and protected source details never appear in context, output, or cached responses. |
| Deleted source or revoked access | The affected information becomes unavailable in the required scope after propagation. |
| Prompt injection inside a source | Retrieved text cannot authorize new actions or override the application’s boundaries. |
| Retrieval outage | The application reports a controlled failure rather than inventing a documented answer. |
Measure retrieval quality
Use relevance judgments for representative queries. Elastic’s ranking-evaluation reference defines common measures such as precision, recall, and reciprocal rank. Two useful starting points are:
- Precision@k: the fraction of the first k results judged relevant.
- Recall@k: the relevant results retrieved in the first k, divided by all relevant results for that question.
Specify whether you judge documents or chunks, the value of k, and how unjudged results are handled. If your relevance labels cover only a limited pool, describe recall relative to that judged pool rather than implying complete knowledge of every relevant passage in the corpus.
Measure answer quality and user outcomes
Separate these questions: is the answer correct, is it supported by the retrieved context, does it cover the necessary information, and do its citations support its claims? AWS’s RAG evaluation metrics treats correctness, faithfulness, completeness, and citation measures as distinct dimensions.
An answer can faithfully repeat an outdated document and still be wrong for the user. Review source applicability as well as grounding. Check both unsupported claims and useful answers the assistant unnecessarily refuses. Use human review to calibrate automated judgments, especially for consequential questions.
Track end-to-end latency, errors, cost per completed task, and whether users can finish the intended workflow. Record the retrieval configuration, source versions, model, and prompt version for a run so you can investigate regressions. Choose release thresholds for the task’s risk and your baseline; no single score proves a RAG system is ready for every use case.
Diagnose Problems Before Adding More Complexity
Inspect the evidence passed to the model before changing prompts or models. Use the following troubleshooting sequence:
- The answer is absent from the sources: assign a content owner to fill the gap.
- The source exists but was not retrieved: inspect parsing, filters, query wording, and retrieval ranking.
- The retrieved passage lacks a condition: inspect chunk boundaries and surrounding context.
- The context is correct but the answer is not: review conflicting passages, answer instructions, and generation behavior.
- The answer uses an obsolete policy: inspect source authority, effective dates, synchronization, and caches.
- The answer is slow or expensive: measure time and usage by stage before reducing context or removing a retrieval step.
Agentic retrieval can decompose a complex question and perform additional searches. Consider it when your evaluation shows that a simpler retrieval path cannot assemble the needed evidence. Add limits and traces so repeated retrieval remains observable. Greater architectural complexity should solve a demonstrated problem.
Choose a Platform and Plan the Operating Cost
Choose a platform by the responsibilities it takes on and the controls it exposes. For example, AWS distinguishes managed and customer-managed knowledge bases by who operates ingestion, storage, indexing, and retrieval infrastructure. That is a useful purchasing lens, not proof that one approach is always cheaper or more accurate.
During a platform evaluation, request a demonstration using your own representative documents and questions:
- Does the connector preserve the content structure and metadata your answers require?
- Can the system enforce the actual end user’s permissions and demonstrate revocation behavior?
- Can you inspect retrieved passages, rankings, source references, and failure traces?
- Can you evaluate and change chunking, retrieval, and answer behavior at the level you need?
- Can you export content and metadata, remove derived data, and identify who owns failed synchronization jobs?
Budget for ingestion and operation separately. Ingestion can include parsing, OCR, embeddings, and repeated processing after changes. Serving can include search capacity, storage, reranking, model input and output, and additional retrieval rounds. Microsoft’s cost documentation illustrates why search infrastructure and optional model or enrichment features can create separate charges.
Estimate your workload using corpus size, update frequency, expected traffic, context length, and response length. Add evaluation, monitoring, and content-maintenance effort. Compare equivalent workloads using current provider pricing; a headline subscription price does not establish the total cost of a useful answer.
Launch a Bounded Pilot
Begin with an audience and content collection you can supervise. Before expanding, confirm that:
- Source owners have approved the content and resolved important conflicts.
- Representative questions have reviewed expected answers and supporting sources.
- Permission, deletion, revocation, injection, and outage cases have been exercised.
- Citations lead to accessible evidence, and unsupported questions receive useful limitations.
- A named team owns content corrections, retrieval failures, and model or prompt changes.
- The pilot meets documented quality, latency, and cost requirements for its intended task.
Expand when the evidence supports it. Each new audience, source system, or product version introduces questions to evaluate and ownership to assign. A dependable RAG knowledge base is maintained through that continuing process of source review, retrieval testing, and answer evaluation.
Technical sources reviewed November 22, 2026. Provider documentation is linked beside the relevant claims. The workflow example and pilot checklists are illustrative guidance, not claims of product testing or measured performance.



