Vector Search for Knowledge Bases: A Practical Guide

Last reviewed: November 22, 2026.

Vector search for knowledge bases retrieves articles or passages by comparing their embeddings with an embedding of the reader’s query. Embeddings are numerical vectors produced by a model to represent features of the input, including semantic relationships. Vector search can connect “I changed phones and cannot approve sign-in” with an article titled “Recover access to multi-factor authentication,” even when the wording differs. The embedding model, source content, and retrieval configuration determine whether that connection is useful.

Start by testing a concrete problem: readers cannot find an existing answer because they describe it differently from your documentation. Compare vector retrieval with your current keyword search and a hybrid approach. Keep exact identifiers, access permissions, and content freshness in the evaluation from the beginning.

You can use vector search to return ordinary article links. In retrieval-augmented generation (RAG), a generative model uses retrieved evidence as context for its response. That answer-generation layer is optional: Amazon Bedrock’s retrieval documentation, for example, separates retrieving source material from retrieving it and generating a response. That distinction lets you improve discovery before introducing a chatbot.

Choose a retrieval method for the questions readers ask

A knowledge base receives several kinds of queries. Some describe a problem in everyday language; others contain an exact error code or product name. Use that mix to choose your retrieval strategy.

MethodUseful knowledge-base queriesWhat to watch
Keyword searchError codes, named features, product identifiers, and wording that appears in the article.Vocabulary differences can hide useful content. Configure tokenization, synonyms, and exact-value fields where appropriate.
Vector searchParaphrased questions and conceptual descriptions of a problem.A related passage can still describe the wrong version, audience, or procedure. Similarity does not establish correctness.
Hybrid searchCollections that receive both natural-language questions and exact-term queries.The combination needs evaluation and tuning; adding another retrieval signal does not guarantee better results.

BM25 is a common keyword-ranking method that uses term matches, term rarity, and document length. In Microsoft’s hybrid search implementation, full-text and vector queries run in parallel and their results are combined. Its documentation specifically identifies codes, specialized terms, dates, and names as reasons to retain keyword matching.

Microsoft diagram combining keyword and vector retrieval through reciprocal rank fusion and optional semantic reranking.

For an exact identifier such as the fictional error code KB-204, test the actual analyzer and query behavior. Full-text search can split or normalize identifiers; an exact-match field or query may be necessary. For an ambiguous query such as “reset access,” offering a clarification or several clearly labeled results can be more useful than guessing which procedure the reader needs.

Build the ingestion and query paths separately

The ingestion path prepares content before a reader searches. The query path finds relevant evidence when a request arrives. AWS’s knowledge-base architecture illustrates the core sequence: parse documents, divide them into chunks, create embeddings, and keep a mapping from the indexed content to its original source.

  1. Ingest: Select approved sources, extract their content, split it into useful retrieval units, and attach source and permission metadata.
  2. Index: Generate document embeddings and store searchable text, vectors, and metadata in the chosen retrieval system.
  3. Retrieve: Establish the reader’s authorized scope, encode the query, and obtain candidates through vector search, keyword search, or both.
  4. Rank and deliver: Combine or rerank eligible candidates, remove unnecessary duplicates, and return linked passages or pass the selected evidence to an answer model.

Keep the approved source repository authoritative. Treat its search index as a reproducible representation that can be rebuilt. A successful upload should be followed by a retrieval check: confirm that the expected section, source link, revision, and access scope appear correctly.

Prepare chunks that can answer a question

Check extraction before tuning search

Inspect the text that actually enters the index. Navigation menus, repeated footers, broken table rows, and missing scanned text can produce poor evidence before ranking begins. A file extension alone does not establish that a parser preserved its contents.

For text-based retrieval over scanned pages, use an extraction path that can recover their text. For tables, figures, or visually meaningful layouts, check whether text extraction is sufficient or whether multimodal processing is needed. AWS’s parsing guidance distinguishes its text-output parser from options that process visual elements. Examine representative extracted pages before committing to a parser.

Keep scope, prerequisites, and warnings together

Choose a retrieval unit that contains a usable piece of evidence. A whole article can combine unrelated topics; a tiny fragment can lose the condition that makes its instruction correct. Microsoft’s chunking documentation describes fixed-size, structure-based, semantic, and combined approaches.

For a troubleshooting article, preserve the section heading with its steps and the applicable product or version. Keep a warning with the action it qualifies. When a table must be split, retain the column labels and enough surrounding explanation to interpret each row.

Compare chunking approaches on your actual questions. Respect the embedding model’s input limit and check truncation behavior. Overlap can preserve continuity, but it also repeats evidence and increases indexing work. Select chunk size and overlap through evaluation; a vendor’s starting value is not an optimum for every knowledge base.

Microsoft table comparing fixed-size, variable-size, semantic, and combined chunking approaches alongside document parsing.

Store metadata that connects each chunk to its source

Each chunk needs a stable relationship to its parent document. The following is an illustrative metadata record, not a request format for a particular service. The indexed passage text and vector would be stored alongside it.

{
  "document_id": "kb-account-access",
  "chunk_id": "kb-account-access:replace-phone:rev3",
  "title": "Recover access to multi-factor authentication",
  "section_path": [
    "Account access",
    "Replace your phone"
  ],
  "source_url": "https://help.example.com/account-access#replace-phone",
  "source_revision": "3",
  "tenant_id": "example-organization",
  "locale": "en-US",
  "product_version": "2",
  "status": "published",
  "allowed_group_ids": [
    "employees"
  ],
  "embedding_profile": "kb-english-v1"
}

Use source identity to replace or remove every child chunk when an article changes. Populate permission fields through trusted ingestion and authorization logic. An allowed_group_ids field is useful only when the application validates its origin and enforces it. Microsoft’s document-level access guidance explicitly distinguishes trusted indexed permissions from ordinary document content.

Keep query and document embeddings compatible

Document vectors and query vectors must be comparable within the intended embedding space. Record the model and revision, vector dimensions, query and document task settings, preprocessing, and any normalization. Follow the model’s intended query/document modes rather than assuming the two inputs always use identical settings.

The distance metric must also match the embedding configuration. Cosine similarity compares vector direction, dot product also depends on magnitude, and Euclidean distance measures separation. Services may transform those values into their own ranking scores; Microsoft’s vector-ranking documentation shows one such transformation. A score is a ranking signal, not the probability that an answer is correct.

Treat a change of embedding model as a migration. Matching vector dimensions alone does not make old and new representations compatible. Microsoft’s migration guide covers separate replacement vectors and coordinated changes to indexing and query configuration. Rebuild the affected embeddings, test the new configuration, and switch the query path only when it matches the new index. Keep a rollback path.

Combine candidate retrieval, fusion, and reranking deliberately

These stages solve different problems. Candidate retrieval finds potentially useful passages. Fusion combines results from multiple retrieval methods. A reranker evaluates a smaller candidate set to improve its final order.

Reciprocal Rank Fusion (RRF) combines positions in ranked lists, allowing lexical and vector results to contribute without treating their raw scores as equivalent. It is one fusion method. Microsoft documents RRF followed by optional semantic reranking; its ranking overview illustrates the stages in that implementation.

A cross-encoder reranker scores the query and candidate passage together. The Sentence Transformers retrieval-and-reranking guide explains why this more expensive scoring stage is normally applied to a shortlist. A reranker can improve the order of retrieved evidence, but it cannot recover a useful passage absent from that shortlist.

Tune candidate counts separately from the number of results shown to a reader or sent to an answer model. Inspect whether repeated overlapping chunks crowd out other useful sources. For article search, consider grouping by parent article. For generated answers, retain distinct supporting passages within a defined context budget. Increase retrieval or reranking work only when the measured benefit justifies the added latency and cost.

Choose an index and storage approach for your workload

Compare exact search with approximate search

Exact nearest-neighbor search identifies the nearest vectors under the selected metric. Approximate nearest-neighbor search (ANN) reduces search work while potentially missing some of those neighbors. An exact-search baseline is useful when evaluating that tradeoff; neither method guarantees that the nearest passage answers the question.

pgvector supports exact search and approximate HNSW and IVFFlat indexes. HNSW navigates a layered graph; IVFFlat searches selected groups of vectors. The project documents different memory, build-time, and speed–recall tradeoffs. Weaviate’s vector-index guide provides a visual explanation of HNSW’s layers and search path.

HNSW search moving through three graph layers toward a neighborhood of candidate vectors.

Start with the simplest configuration that meets your measured workload. Include real filters, concurrent requests, updates, and cache conditions in the test. A fast unfiltered query on a quiet system does not establish the latency your readers will experience.

A dedicated vector database is optional

ApproachWhen to investigate itWhat to validate
Existing database with vector supportYour data and operational expertise already live there.Index capabilities, filtering behavior, search features, workload interference, and backup/recovery.
Search engine with vector and keyword retrievalYou need article search, lexical controls, and hybrid ranking together.Ingestion, source synchronization, authorization integration, ranking controls, and capacity.
Dedicated vector storeYour retrieval workload benefits from its indexing, filtering, or scaling capabilities.The surrounding pipeline: parsing, lexical retrieval, permissions, freshness, and observability.
Managed retrieval serviceYou want the provider to handle more of ingestion and retrieval.Supported sources and formats, permission behavior, evaluation visibility, customization, and billing.

These are architecture choices, not a universal product ranking. Microsoft’s vector-search service decision guide makes existing infrastructure, update requirements, and operational complexity central considerations. Measure those requirements before adding another system.

Enforce access and content scope before returning evidence

Derive tenant and access constraints from authenticated, trusted authorization context. Carry the applicable source permissions into every chunk, and enforce them before text reaches a reader or language model. OWASP’s RAG Security Cheat Sheet emphasizes retrieval-time access checks and permission inheritance. Asking the answer model to conceal restricted text is insufficient.

Apply the same authorization policy to keyword retrieval, vector retrieval, fallback paths, and cached responses. Check permission changes after ingestion. A search interface that hides a restricted result after its text has already entered the model context has applied the control too late.

OWASP checklist for per-chunk permissions, retrieval-time access checks, and tenant isolation.

Separate authorization correctness from the search engine’s internal filtering strategy. Microsoft’s vector-filter documentation explains how prefiltering and postfiltering affect candidate selection and recall. A selective postfilter can return too few results even when eligible documents exist. Test the engine’s behavior under your real access and product filters while keeping access enforcement mandatory.

Also treat retrieved text as untrusted input. A document can contain instructions that attempt to redirect an answer model. Preserve source provenance, constrain tool permissions, and keep access checks outside the model. Embedding content does not make it safe or authoritative.

Keep edits, removals, and permission changes in sync

Define freshness as an observable behavior: how long does a source change take to appear in the search experience? An ingestion job’s completion state and the availability of updated results can be separate milestones.

For example, Amazon Bedrock’s sync documentation describes incremental handling of new, changed, and deleted documents. It also documents conditions under which metadata-only changes can reuse embeddings, and notes that availability after ingestion depends on the vector store. Verify the behavior of your actual connector and configuration.

Maintain document-to-chunk mappings and a source revision or content hash. When boundaries change, remove obsolete child chunks as well as adding replacements. A permission-only change may need a metadata update without a new embedding, but that depends on the pipeline. Coordinate any application caches with the same source and authorization lifecycle.

  • Edit test: Change a distinctive sentence and check when the new revision appears through every active retrieval path.
  • Revocation test: Remove access for one test identity and verify that its searches and cached answers no longer expose the document.
  • Deletion test: Remove or retire a test document and verify that its child chunks and cached evidence are no longer served.
  • Migration test: Confirm that query encoding, document vectors, and cached embeddings use the intended configuration after a model change.

Define acceptable propagation times before launch and monitor failures against them. For time-sensitive access revocation, the design may need an authoritative permission check or a deny rule while index synchronization completes.

Evaluate retrieval before judging generated answers

Build a test set around real failure modes

Use representative questions from search logs, support requests, and content owners. Remove personal or sensitive information that is unnecessary for evaluation. For each question, identify the relevant source passages, allowed scope, and expected behavior. Keep some queries separate from the set used to tune the system.

The following cases describe a fictional support knowledge base. They are proposed tests, not measured search results.

Query or changeWhat a successful test should establish
“I changed phones and cannot approve sign-in.”Retrieves the applicable account-recovery passage despite different article wording.
“What does KB-204 mean?”Finds the exact code’s article and does not confuse a similar identifier.
“Reset access.”Handles ambiguity with useful choices or clarification.
“What is the upload limit in version 2?”Uses the requested version and excludes retired guidance from the answer context.
An employee requests an administrator-only procedure.Restricted evidence never reaches that user or the answer model acting for them.
A source is edited, restricted, or deleted.The active search and cache paths reflect the intended lifecycle.
A question asks about a policy absent from the corpus.The application acknowledges insufficient evidence or offers an appropriate next step.

Measure three different layers of quality

LayerQuestionUseful measurement
ANN neighbor recoveryDid the approximate index recover the neighbors exact search would find?Compare exact and ANN results with the same vectors, metric, eligible corpus, and filters.
Evidence retrievalDid the returned passages address the reader’s information need?Relevance judgments, precision/recall at k, MRR, or nDCG.
Generated answerDid the response use the evidence faithfully and address the question?Groundedness, completeness, correctness, and citation support, assessed separately.

A perfect match to exact nearest neighbors can coexist with poor human relevance. Likewise, an answer can accurately repeat an outdated source. Microsoft’s RAG evaluation documentation separates retrieval evaluation, groundedness, relevance, and completeness. Model-based judges can help review outputs, but their scores need calibration and human checks.

Microsoft documentation separating retrieval process evaluation from final-response system evaluation.

Use metrics that match the search experience

Elastic’s ranking-evaluation reference defines several useful measures. Precision@k measures the relevant share of the top results; recall@k measures how much of the known relevant set was retrieved. MRR rewards placing the first relevant result early. nDCG can account for both rank and graded relevance.

For an illustrative ranking of five results labeled irrelevant, relevant, relevant, irrelevant, relevant, precision@5 is 3/5, or 0.60. If four relevant documents are known for that query, recall@5 is 3/4, or 0.75. The first relevant result is second, so reciprocal rank is 1/2, or 0.50. MRR averages reciprocal rank across queries. These numbers explain the calculation; they are not a product benchmark.

Specify whether you are evaluating articles or chunks. Multiple overlapping chunks from one article can distort the impression of coverage. Recall also depends on the completeness of your relevance labels; an unlabeled useful result should trigger review of the test set.

Compare configurations fairly

Run keyword, vector, hybrid, and optional hybrid-plus-reranking configurations on the same corpus snapshot, test queries, and authorization scope. Use a consistent reporting cutoff. For generated-answer comparisons, also control the final evidence budget and answer-model configuration so that a retrieval change is not confused with a larger prompt or a different generator.

Record latency distributions, including p95, under representative concurrency, along with failure rates and cost. Review results by query class: an overall average can hide missed error codes or degraded access-scoped searches. Set acceptance criteria before looking at the comparison, with unauthorized disclosure treated as a release blocker.

Diagnose the stage that caused a bad result

Use these as starting hypotheses, then inspect the source text, selected chunks, filter decisions, and intermediate rankings.

Observed problemFirst checks
The correct article never appears.Was it ingested and searchable? Is its text intact? Do the query and document embeddings match? Did a filter remove it?
The topic is right but the instruction is wrong.Check chunk scope, product/version metadata, retired content, and whether the retrieved passage actually answers the question.
Exact codes are missed.Inspect tokenization, exact-value fields, lexical candidates, and fusion behavior.
Search returns too few eligible results.Inspect filter selectivity, ANN search effort, candidate counts, and the engine’s filtering mode.
The same article fills the results.Check overlapping chunks, duplicate sources, parent-document grouping, and final evidence selection.
Retrieved evidence is correct but the generated answer is wrong.Inspect the final context and generator behavior separately from retrieval. Check citation support and conflicting passages.
Old or restricted content reappears.Trace synchronization, obsolete child chunks, cached results, permission changes, and fallback paths.

Estimate the cost of the complete pipeline

Budget for the workflow you will operate, including updates. The initial vector-store bill is only one component. Microsoft’s cost-planning documentation illustrates how search capacity or consumption, storage, enrichment, model calls, and optional ranking features can be charged separately.

  • Preparation: Connectors, document parsing, OCR or multimodal extraction, and initial embedding generation.
  • Storage and indexing: Passage count, vector dimensions and representation, index structures, metadata, replicas, and backups.
  • Requests: Query embeddings, retrieval work, reranking, and optional answer generation.
  • Maintenance: Re-embedding changed content, model migrations, synchronization, monitoring, relevance reviews, and incident handling.

Measure a representative sample and normal plus busy-period traffic, then apply the provider’s current billing units. Include the effect of chunk overlap and repeated indexing. Compare the operating effort and quality you actually need rather than choosing from storage price alone.

Start with a bounded pilot

Choose one useful collection and one reader workflow. Establish its keyword baseline, add compatible embeddings, and evaluate hybrid retrieval where it addresses actual query failures. Before wider rollout, confirm that:

  • Representative questions retrieve useful, correctly scoped evidence.
  • Exact identifiers and ambiguous queries receive appropriate handling.
  • Unauthorized and retired content stay out of results and generated context.
  • Edits, deletions, and permission changes propagate within the required interval.
  • Latency and operating cost remain acceptable under the intended workload.
  • Content ownership, evaluation, and index maintenance have named responsibilities.

Choose the retrieval configuration that meets those requirements with manageable complexity. Add generated answers only when the selected evidence is dependable and the answer layer passes its own evaluation.