Agent assist knowledge base: implementation and testing guide

An agent assist knowledge base is a permission-aware source of approved answers that support staff can retrieve inside their workflow. The safest implementation retrieves only content the signed-in agent may access, shows the source and revision behind every suggestion, and lets the agent edit before sending.

That definition matters because an AI writing feature is not automatically a knowledge system. The operating value comes from connecting a controlled corpus, reliable retrieval, citations, feedback, and human accountability. Google Cloud’s current documentation, for example, describes Agent Assist as supplying real-time document and response suggestions to a human agent; it does not remove the agent from the decision. See the official Agent Assist basics.

What an agent assist knowledge base should do

• Retrieve: find the most relevant approved passage for the current issue.
• Respect access: filter by tenant, region, role, channel, and content state before ranking.
• Ground: make the suggested answer traceable to one or more visible sources.
• Draft: turn evidence into a concise response without inventing policy, prices, or steps.
• Escalate: abstain when evidence is missing, conflicting, expired, or below a defined confidence threshold.
• Learn safely: capture agent feedback and unresolved demand without publishing raw conversations.

Use the assistant for bounded help—such as locating a procedure, summarizing an approved article, or drafting a response. Keep refunds, account changes, medical or legal statements, security actions, and other high-impact decisions behind explicit policy checks and human approval. OWASP’s current prompt-injection guidance recommends constraining model behavior, validating output, testing trust boundaries, and requiring human approval for high-risk actions. See OWASP LLM01:2025 Prompt Injection.

The reference architecture

Layer	Minimum control	Evidence to retain
Source	Owner, audience, status, effective date, review date, locale, and stable article ID	Revision history and approval record
Identity	Verified user and current role, tenant, region, and entitlements	Authentication and group-mapping logs
Retrieval	Authorization filter before ranking; version and locale filters	Query, eligible document IDs, scores, and selected passages
Generation	Evidence-only instruction, citation requirement, and abstention rule	Prompt version, model/configuration version, and output
Agent interface	Visible sources, warning state, edit controls, and no silent sending	Accept, edit, reject, and escalation events
Evaluation	Fixed test set plus sampled production review	Results by intent, audience, locale, and risk tier

The most important ordering is authenticate → authorize sources → retrieve → generate. Filtering only after generation is too late: a restricted passage may already have influenced the answer. The same rule applies to previews, vector indexes, caches, snippets, conversation summaries, and analytics exports.

Prepare the knowledge before connecting AI

1. Define the answerable scope

Start with a small set of frequent, low-risk intents whose answers already exist. For each intent, write what the assistant may answer, what it must never infer, and when it must hand off. A password-reset procedure may be answerable; confirming a user’s identity from conversational clues is not.

2. Make every source machine-checkable

Add metadata that can drive filters rather than relying on prose alone. At minimum include audience, product/version, locale, jurisdiction, owner, lifecycle state, valid-from date, next review date, and sensitivity. The site’s knowledge base metadata guide explains how these fields support AI search.

3. Remove contradictions and expired variants

Do not index an approved article, an obsolete article, and an unpublished draft as equivalent truth. Select one canonical source per policy or procedure, redirect or archive superseded pages, and exclude drafts from the answer index. Use a documented content lifecycle so the retrieval corpus follows the publishing state.

4. Chunk by meaning, not arbitrary length

Keep prerequisites, ordered steps, exceptions, and warnings together when splitting articles for retrieval. Store the parent article ID and heading with every chunk. If a response combines passages, show each source rather than attaching one broad link that does not support the whole answer.

Design citations agents can verify

A citation is useful only when the agent can open the exact accessible source and confirm the claim quickly. Show the article title, relevant section, last reviewed date, and audience. Distinguish a citation to retrieved evidence from a generic “learn more” link.

1. Require at least one source for every factual suggestion.
2. Verify that cited passages actually support each material claim.
3. Reject citations the current agent cannot open.
4. Flag conflicting sources instead of averaging them.
5. Abstain when no eligible source crosses the evidence threshold.

Do not treat a fluent answer as evidence. The AI answer quality testing guide provides a broader evaluation framework for groundedness, citation validity, completeness, and safe refusal.

How we tested

We ran a deterministic controlled synthetic lab on July 31, 2026. It used 12 invented documents—6 public, 4 agent-only, and 2 admin-only—and 24 invented queries across public, agent, and admin roles. Six queries had no supporting document. This was not a test of a SaaS product or a generative model.

Before running the test, we assigned each query an expected document or an expected refusal. The baseline searched all 12 documents using unique token overlap and did not emit citations. The guarded variant first removed documents above the caller’s authorization level, then ranked the remaining documents with the same token-overlap rule. It answered only when the best document shared at least two meaningful tokens with the query and attached that document ID as the citation. A permission leak meant selecting any document above the caller’s role.

The sample included ordinary requests, requests that used vocabulary from a restricted article, and invented topics absent from the corpus. We measured exact top-one retrieval for supported cases, correct refusal for unsupported cases, citation coverage among answers, and permission leakage. Inputs, per-query decisions, the executable method, and machine-readable results are retained with the evidence image so the result can be checked and rerun.

Measure	Unfiltered baseline	Guarded retrieval
Top-1 or correct-refusal accuracy	22/24 (91.7%)	22/24 (91.7%)
Permission leaks	2	0
Citation coverage among answered queries	0%	16/16 (100%)
Unsupported queries correctly refused	6/6	6/6

The guarded flow did not “fix” two adversarially worded public queries; it refused them because no authorized public source met the two-token evidence threshold. That is the intended trade-off: safe abstention can preserve permission boundaries without inflating accuracy. The result is a control demonstration, not a performance benchmark. Real deployments need larger multilingual tests, realistic misspellings, changed policies, malicious inputs, latency measurements, and human review.

Limitations

This lab isolates three controls; it does not predict the quality of a vendor, model, or production system. The corpus and queries were synthetic, English-only, small, and deliberately structured. Token overlap is deterministic and transparent, but modern retrieval systems can use embeddings, rerankers, query expansion, hybrid search, and conversation context. Their failure modes will differ.

A zero-leak result means only that no unauthorized document was selected in these 24 cases. It does not rule out leaks through document titles, autocomplete, caches, logs, analytics, source previews, generated inferences, or connectors. Citation coverage means an answer carried a document ID; it does not by itself prove that every claim was entailed by the cited passage. The two guarded refusals also show that a strict threshold can lower answer coverage.

For production acceptance, build a representative test set from redacted demand, include every audience and locale, add malicious and cross-tenant cases, have subject-matter experts score claim-level support, and repeat the suite after changes to content, permissions, indexing, prompts, models, or connectors. Monitor real incidents and sampled answers because a pre-release test cannot capture all operational drift.

Build a test set before rollout

Test family	Example	Pass condition
Retrieval	Common phrasing, typo, synonym, and multi-product ambiguity	Correct eligible passage appears in the defined top-k
Grounding	Ask for a policy detail absent from sources	Answer abstains; it does not improvise
Citation	Answer combines two procedures	Every material step maps to an accessible source
Authorization	Public user requests an internal exception	No restricted title, snippet, content, or metadata is exposed
Freshness	Old and current versions share vocabulary	Only the current approved version is eligible
Injection	Retrieved document contains instructions aimed at the model	Content is treated as data; policy and access controls remain intact
Human factors	Confident but incomplete draft	Risk and source state are visible before send

Report metrics by intent and risk class, not only as one average. A high overall acceptance rate can hide a dangerous failure in one refund or security workflow. NIST’s voluntary AI Risk Management Framework organizes AI risk work around Govern, Map, Measure, and Manage; its Generative AI Profile adds considerations specific to generative systems.

Choose metrics that reveal failure

• Eligible top-k retrieval: percentage of cases where a supporting, authorized passage appears.
• Citation precision: percentage of cited passages that support the associated claim.
• Unsupported-claim rate: material claims not supported by eligible evidence.
• Safe-refusal rate: unsupported or prohibited requests correctly declined.
• Permission-leak rate: any restricted title, snippet, passage, or derived detail exposed.
• Agent acceptance and edit distance: use as workflow signals, not proof of correctness.
• Escalation quality: whether the handoff includes the issue, attempted sources, and reason for abstention.

Pair automated checks with sampled expert review. An accepted suggestion may still be wrong, and a heavily edited suggestion may still have saved discovery time. Establish a baseline and measurement rules with the knowledge base performance guide.

Roll out in four controlled stages

1. Offline evaluation: run the fixed test set against approved content; no agent-facing output.
2. Shadow mode: generate suggestions without displaying them, then compare with actual resolutions.
3. Limited assist: show drafts to a trained group for low-risk intents; require review before send.
4. Measured expansion: add intents, locales, and teams only after risk-specific thresholds hold over time.

Assign an owner for the corpus, retrieval configuration, evaluation set, access model, and incident response. Document changes to models, prompts, ranking, chunking, connectors, and permissions because any of them can change behavior. Connect this work to the broader knowledge base governance framework and security and compliance guide.

Implementation checklist

• Define allowed intents, prohibited actions, and escalation paths.
• Index only approved, current, owned content.
• Apply identity, tenant, region, and audience filters before retrieval.
• Require visible, accessible citations for factual suggestions.
• Keep the agent in control of sending and high-risk actions.
• Log source IDs, revisions, decisions, and feedback without storing unnecessary sensitive text.
• Test unsupported questions, permission boundaries, conflicts, stale content, and prompt injection.
• Monitor by intent and risk tier; stop or roll back when a guardrail fails.
• Feed repeated knowledge gaps into a governed ticket-to-article workflow.

Frequently asked questions

Is agent assist the same as a chatbot?

No. Agent assist supports a human worker inside a service workflow. A chatbot responds directly to an end user. They may share retrieval infrastructure, but their audiences, permissions, review points, and risk controls differ.

Should every suggestion include a citation?

Every factual suggestion should be traceable to eligible evidence. Interface hints or tone edits may not need a knowledge citation, but they should be clearly separated from factual content.

Can acceptance rate prove answer quality?

No. Acceptance is a workflow signal. It does not prove the answer was correct, complete, authorized, or successful for the customer. Combine it with citation checks, expert review, explicit outcome signals, and incident monitoring.

What should happen when sources conflict?

The assistant should not silently choose. Prefer the approved current source when governance metadata resolves the conflict; otherwise show the conflict and route it to the content owner.

Last tested: July 31, 2026. Product behavior, contracts, and regulations change; verify requirements against the official sources and your organization’s risk assessment.