Skip to content

fix(post): send LlmVerifier requests to a chat completions endpoint - #699

Open
feiiiiii5 wants to merge 1 commit into
zilliztech:mainfrom
feiiiiii5:fix/llm-verifier-chat-endpoint
Open

feiiiiii5 wants to merge 1 commit into
zilliztech:mainfrom
feiiiiii5:fix/llm-verifier-chat-endpoint

Conversation

@feiiiiii5

Copy link
Copy Markdown

Problem

LlmVerifier / llm_semantic_verification never returns a verified answer. The function builds a
chat request — it sends messages= and reads resp.choices[0].message.content — but then sends
it to the completions endpoint:

client = client if hasattr(client, 'completions') else client.chat
resp = client.completions.create(model=..., messages=[...], ...)

completions.create takes prompt, not messages, so the call raises; the bare except Exception
turns that into one printed line and return None. Because post_process_messages_func=None is what
tells the adapter to miss the cache, the verifier silently degrades into "always call the real LLM" —
the feature added in #669 does nothing, for any client shape.

The repository's own unit test already encodes the intended contract (a client.chat.completions
object) and is red on main:

$ python3 -m pytest tests/unit_tests/processor/test_post.py -q
FAILED tests/unit_tests/processor/test_post.py::test_llm_verifier - AssertionE...
1 failed, 4 passed

The hasattr(client, 'completions') probe also picks the wrong branch for a Mock (attributes are
auto-created) and for openai>=1.0 (which has both chat and completions), and it never matches
the pre-1.0 module-level ChatCompletion that examples/processor/llm_verifier_example.py uses.

Fix

Route the chat-shaped request to a chat endpoint, keeping both client generations the docs use:

  • client.chat.completions.create when the client exposes .chat (openai>=1.0, langchain-style clients);
  • client.ChatCompletion.create for the legacy module client;
  • client.completions.create only as the last resort, as before.

No behaviour changes for callers, and the except is left in place so a real API error still degrades
to a cache miss rather than raising inside the post-processor.

Verification

Same test file, two trees, base pinned at c59fb3a6152a4458b2a070ca183b61c4b614095f:

### base tree source is pristine upstream
$ git -C base-wt diff -- gptcache/processor/post.py     # empty

=== BASE (upstream source + this PR's tests) ===
3 failed, 5 passed in 0.75s
  test_llm_verifier                                     # the repo's existing test
  test_llm_verifier_uses_the_chat_endpoint              # added here
  test_llm_verifier_supports_legacy_module_client       # added here
AttributeError: '_LegacyClient' object has no attribute 'chat'   # post.py:144

=== HEAD (this fix + the same tests) ===
8 passed in 0.54s

cwd /private/tmp/r16-parent/{base-wt,gptcache}, PYTEST_DISABLE_PLUGIN_AUTOLOAD=1,
python -m pytest tests/unit_tests/processor/test_post.py -q -o addopts="", base rc=1 / head rc=0,
and gptcache.processor.post.__file__ was printed in each run to prove the base run imported the
upstream copy, not the patched one.

New tests use hand-written fakes instead of a bare Mock so the endpoint actually under test is
observable: _ChatClient sets completions = None, so calling it raises instead of silently
returning a Mock, and _LegacyClient has no .chat at all.

Two notes so nobody chases a phantom: tests/pytest.ini adds --html=... --self-contained-html, so
a bare pytest errors unless pytest-html is installed or -o addopts="" is used; and
tests/unit_tests/processor/test_context.py fails to collect in my local venv with
openai.lib._old_api.APIRemoved because that venv has openai>=1.0 — that is an environment
artifact, not something this patch touches.

Checklist

  • bug fix, and the user-visible effect ("semantic verification silently never fires") is stated above
  • tests added to the existing unit test module, no existing assertion weakened
  • git diff limited to gptcache/processor/post.py and tests/unit_tests/processor/test_post.py

@sre-ci-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: feiiiiii5
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The verifier builds a chat request (messages= plus choices[0].message.content)
but sends it to client.completions.create, which takes prompt instead. The
resulting error is swallowed by the bare except, so semantic verification
always returned None and every hit fell through to a real LLM call. Route the
request to client.chat.completions (openai>=1.0) or ChatCompletion (legacy
module client), which is what the existing unit test and the example expect.

Signed-off-by: feiiiiii5 <204683769+feiiiiii5@users.noreply.github.com>
@feiiiiii5
feiiiiii5 force-pushed the fix/llm-verifier-chat-endpoint branch from 0c18da1 to e37f0e7 Compare September 20, 2026 08:07
@mergify mergify Bot added dco-passed and removed needs-dco labels Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants