Investigate a weak knowledge answer

A field investigation

Do not “add more knowledge” until you know what failed.

A weak answer can come from the source, synchronization, retrieval, runtime permissions, instructions, or the question itself. Fixing the wrong layer creates clutter and hides the real problem.

Begin with one failed question

Copy the exact visitor question into a safe test record. Write the answer you expected, the source that authorizes it, and why that source—not another page—is correct. If you cannot name an authoritative source, the Knowledge Base is not yet the problem; your organization needs an approved answer.

Start a new conversation for the repeat test. Previous turns can shape the model’s interpretation and make a content change look better or worse than it is. Use the same Assistant/runtime, language, and conditions so the comparison is meaningful.

Confidence is a clue, not a truth meter

Knowledge Gaps uses heuristic confidence and offers thresholds from 50% to 90%, with 70% as the default view. A high-confidence answer can still be wrong, and a cautious accurate answer can score lower. Use confidence to find cases for review, then inspect the evidence.

A five-layer diagnosis

Move through these layers in order

Layer	Question to answer	Evidence	Typical repair
1. Authority	Does a current approved source state the answer clearly?	Read the actual page/file and compare duplicates.	Correct or consolidate source content; do not invent metadata.
2. Publication	Is that exact source version available to retrieval?	Selected/Pending/Current state, file Completed state, sync item result.	Save, sync, resolve item error, or replace failed file.
3. Capability	Could this runtime/agent search knowledge?	Connected vector store, Basic Assistant setup, or specialist Knowledge Base setting.	Enable knowledge for the intended specialist or test the right runtime.
4. Retrieval	Did search run and find the intended source with useful relevance?	Chat History knowledge-search event, query, files and scores when available.	Improve title/source wording and concise metadata; remove confusing duplicates.
5. Response	Did the assistant use the retrieved evidence correctly?	Retrieved snippets versus final answer and Assistant/Agent instructions.	Clarify response rules, source priority, uncertainty behavior, or specialist scope.

This order prevents a common mistake: adding twelve keywords when the expected PDF is still processing, or resyncing ten times when the specialist is not allowed to use the Knowledge Base.

Read Chat History like a detective

Open the failed conversation in Analytics. Look for whether knowledge search was triggered, the search query recorded by the runtime, which files or page documents were found, relevance scores when provided, tool activity, active Assistant or specialist, and the final response.

• No search event: the runtime may not have treated the question as a knowledge need, the vector store may be unavailable, or the active advanced specialist may have knowledge disabled.
• Search ran, wrong source found: sources may overlap, titles may be vague, or visitor terminology may be absent.
• Right source found, poor answer: compare the retrieved passage with the final response and review instructions, contradictions, and missing qualifiers.
• Right source absent: verify selection/current state or Completed indexing before rewriting anything.

Scores are relative signals supplied by retrieval, not a universal pass mark. Compare them within repeated tests rather than publishing a rule that “anything above X is true.”

Three cases, from symptom to fix

Case 1: the visitor says “returns window”

The approved page is titled “Right of Withdrawal.” Search runs but the page receives weak relevance. Add the real visitor phrase to Topics and a concise FAQ grounded in the policy, save, sync, and repeat in a new chat. This is a vocabulary bridge: the source was correct, publication worked, and retrieval needed clearer language.

Case 2: the old price keeps appearing

The current pricing page is Current, but an old brochure PDF is also Completed. Search sometimes finds the brochure. Do not add “NEW PRICE” keywords to compete. Remove the obsolete file, verify the approved page explicitly states currency, market, tax basis, and effective date, then repeat the test.

Case 3: the appointments specialist never searches

Basic runtime answers from the policy, while Advanced runtime’s appointments agent does not. The shared Knowledge Base is healthy. Inspect that specialist’s Knowledge Base field and published workflow revision. Enable access where appropriate, publish the flow change, then use the flow simulator and a new conversation. This was a capability boundary, not a sync issue.

The improvement loop

1. Freeze the case. Keep the exact question, expected answer, source, runtime, and timestamp.
2. Inspect the trace. Determine whether search ran and which source was retrieved.
3. Fix one layer. Change the source, publication state, capability, metadata, or instructions—not all of them together.
4. Publish correctly. Save and Sync page changes; wait for file indexing; publish an Agent Flow revision when specialist settings changed.
5. Open a fresh chat. Repeat the original wording first, then one natural variation.
6. Compare evidence. Record retrieval source, answer accuracy, scope, and whether uncertainty was handled honestly.
7. Keep or revert editorially. Retain a change only if it improves the intended case without pulling unrelated questions toward the source.

Symptom map for fast triage

What to check before escalating

Symptom	First check	Do not jump straight to
Page says Pending	Run Sync and inspect its item result.	Rewriting metadata.
File says processing	Wait for Completed or terminal failure.	Calling the source inaccurate.
Search never triggered	Connected store, active runtime, specialist knowledge permission, and question intent.	Uploading duplicates.
Wrong source wins	Contradictions, duplicate versions, vague titles, and overlapping metadata.	Stuffing more keywords into every page.
Correct source, missing qualifier	Whether the retrieved passage and concise facts preserve the condition.	Adding an unsupported FAQ.
Good answer in old chat, bad in new chat	Prior context may have supplied facts or routing cues.	Assuming synchronization randomly reverted.
Gap confidence looks low	Read signals, retrieved sources, and factual accuracy.	Treating the percentage as a model certainty score.

Measure what matters to a visitor

The success test is not “the intended file had the highest score.” It is that the assistant gives an accurate, appropriately scoped answer, preserves important conditions, admits when approved knowledge is missing, and guides the visitor to a sensible next step. Retrieval evidence helps explain the result; it is not the customer outcome itself.

Build a small regression set of real questions: direct wording, synonym wording, an edge case, and a question the source should decline. Run it after substantial source or workflow changes. Ten representative questions teach more than a hundred random chats.

Things that usually make the library worse

• Duplicating the same answer into many pages and files.
• Writing fictional FAQs purely to force retrieval.
• Keeping obsolete documents “just in case.”
• Adding broad Topics that attract unrelated questions.
• Treating Exclude Content as privacy control.
• Changing source, metadata, instructions, and runtime simultaneously so nobody knows which change worked.
• Reviewing only a successful happy-path question.

The evidence panel

knowledge-improve-troubleshoot

Screenshot capture pending /images/screenshots/knowledge/improve-troubleshoot.webp

Improve Answer Quality and Troubleshoot Knowledge

Capture

Open one sanitized low-confidence gap detail with the score, signals, and searched-file section expanded.

Show

Question, confidence, reason signals, searched files and scores, response excerpt

Viewport

Desktop, 1440 × 900

Annotate

Use numbered callouts only for controls referenced in the procedure.

Redact

OpenAI keys, tokens, secrets, personal information, private URLs, IP addresses, and conversation text

Follow the layer you found

• Source language or exclusions: edit page metadata
• Publication mismatch: read sync and status outcomes
• Document state or duplicates: manage the file library
• Specialist capability: write specialists and choose knowledge access