Problem/Motivation

When a user asks an off-topic question (e.g., "Who is Taylor Swift?"), the LLM correctly responds with a refusal message ("I can only answer questions
related to our content library..."), but irrelevant RAG sources are still displayed below the response.

This happens because the current [NO_SOURCES] opt-out mechanism relies on the LLM remembering to append a marker when it does not use
sources. LLMs do not reliably follow this instruction, so when the marker is forgotten, irrelevant sources are shown.

Proposed resolution

Flip the marker logic from opt-out to opt-in:

  • Before (fragile): LLM must append [NO_SOURCES] when it does not use context. Forgotten marker = irrelevant sources
    shown.
  • After (robust): LLM must append [SOURCES_USED] when it does use context. Forgotten marker = sources hidden.

This is a safer default because hiding sources when they should be shown is a minor UX issue, while showing irrelevant sources on a refusal is confusing and
misleading.

Changes to ChatOrchestrator:

  • System prompt instruction updated to require [SOURCES_USED] opt-in
  • parseSourceRelevance() inverted: marker present = sources relevant, absent = not relevant
  • Unit tests updated to match new logic (5 tests, all passing)

Steps to reproduce

  1. Go to /ai-search/chat
  2. Ask an off-topic question like "Who is Taylor Swift?"
  3. LLM responds with refusal message
  4. Irrelevant sources (from supplementary search) are displayed below the response

Remaining tasks

Review and merge.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

paulsheldrake created an issue.

  • paulsheldrake committed 8c6096ca on 1.x
    Issue #3586605: Replace opt-out [NO_SOURCES] marker with opt-in [...