[CRITICAL] LiteLLM discovery: resolveLiteLLMApi routes every OpenAI-tagged model group to /v1/responses; LiteLLM only guarantees /v1/chat/completions

[AincradBot]

[CRITICAL] LiteLLM provider discovery misroutes models to OpenAI Responses API (/v1/responses)

Description

Since 17.4.1 (introduced by merged PR #9085 fix(catalog): route LiteLLM OpenAI models through Responses), the LiteLLM discovery path assigns the client wire API from the model's upstream provider type. Any LiteLLM-discovered model whose rich metadata reports providers: ["openai"] (or litellm_params.custom_llm_provider == "openai", or a backend model/base_model prefixed openai/) is mapped to api: "openai-responses", so OMP sends requests to {baseUrl}/v1/responses.

LiteLLM is a Chat-Completions-compatible gateway: its documented OpenAI-compatible surface is /v1/chat/completions (it transcodes OpenAI-format requests to whatever the upstream speaks). A standard LiteLLM proxy does not guarantee a working /v1/responses endpoint for these models, so affected models fail on every request.

Verified against main (packages/catalog/src/provider-models/openai-compat.ts):

// classifyLiteLLMApiRoute (entry from /model_group/info, /v2/model/info, ...)
if (providers) {
    return providers.every(provider => provider === "openai") ? "openai" : "other";
}
// ...

export function resolveLiteLLMApi(entry, id, fallbackApi = "openai-completions"): Api {
    return classifyLiteLLMApiRoute(entry, id) === "openai" ? "openai-responses" : fallbackApi;
}

Used in the model builder at

api: options.resolveApi?.(entry, id) ?? options.api,   // mapLiteLLMRichEntry

and wired into both discovery paths (litellmModelManagerOptions and discoverLiteLLMModels in packages/coding-agent/src/config/model-discovery.ts).

Why this is wrong: the providers field in LiteLLM's /model_group/info (and custom_llm_provider in litellm_params) describes which upstream provider type LiteLLM uses internally to reach the backend (OpenAI SDK / openai-family integration) — it says nothing about which wire API a client of the proxy should use. This specifically includes model: openai/<anything> pointing at a custom api_base (any OpenAI-compatible server, incl. local/self-hosted and non-Responses backends): LiteLLM still classifies those with the OpenAI provider, so providers == ["openai"] matches far more deployments than "native api.openai.com models". Every LiteLLM model group backed by the openai upstream type is misrouted — potentially a large share of typical LiteLLM deployments — regardless of whether a Responses-capable endpoint exists behind the gateway. (PR review discussion already touched adjacent mixed-backend hazards; the fundamental issue is that the heuristic treats upstream driver as client wire API.)

The unknown-route fallback in the same function (/v1/models fallback path, mapLiteLLMOpenAICompatibleModel → resolveLiteLLMApi(undefined, model.id)) additionally promotes well-known OpenAI model IDs served by any OpenAI-compatible gateway to openai-responses based on the ID alone.

Steps to Reproduce

1. Run a LiteLLM proxy with any OpenAI-upstream model, e.g. config:

   model_list:
     - model_name: gpt-5.6-sol          # or any model
       litellm_params:
         model: openai/gpt-5.6-sol      # custom_llm_provider resolves to "openai"

   (or any gateway exposing /model_group/info where entries carry providers: ["openai"], including openai/… models pointed at a custom api_base).

2. Use omp 17.4.1+ with the litellm provider (LITELLM_API_KEY/stored auth) or a models.yml provider with discovery.type: "litellm".

3. Select the discovered model and send any prompt.

Expected Behavior

LiteLLM-discovered models default to the provider-level API (openai-completions → /{base}/v1/chat/completions), because that is the guaranteed contract of a LiteLLM gateway. The Responses API must not be inferred from the providers/custom_llm_provider field; upstream provider type does not determine the client wire API. If per-model Responses routing is desired for LiteLLM at all, it needs positive evidence that this specific proxy serves /v1/responses (e.g. an explicit config flag such as api: openai-responses, or a capability probe), never a default derived from upstream tags.

Error Output

Every chat request to an affected model is sent to POST {baseUrl}/v1/responses instead of {baseUrl}/v1/chat/completions and fails — 404/Cannot POST /v1/responses on older LiteLLM proxies and non-Responses OpenAI-compatible backends; on proxies that partially implement /responses, downstream errors depending on upstream support. On 17.4.0 the identical deployment works.

Environment

• Platform: any (observed on Windows native; platform-independent)
• omp version: ≥ 17.4.1 (regression from 17.4.0; still present on main and in latest release line 17.4.4)
• Provider: LiteLLM (litellm discovery, and models.yml providers with discovery.type: "litellm")

Workaround (temporary)

• Downgrade to 17.4.0, or
• For each affected model, set modelOverrides in models.yml with an explicit api: openai-completions (impractical with many discovered models).

Suggested direction

Revert resolveLiteLLMApi's promotion of providers == ["openai"] to openai-responses for LiteLLM discovery (both the built-in litellm descriptor and the configured discovery.type: "litellm" path, incl. the /v1/models ID-based fallback), or gate Responses routing behind explicit, opt-in evidence that the proxy actually serves the Responses API. Reasoning-summary support for OpenAI models behind LiteLLM should be handled without breaking the chat-completions default for every other deployment.

[roboomp]

OMP does intentionally choose openai-responses for OpenAI-backed LiteLLM groups in packages/catalog/src/provider-models/openai-compat.ts (classifyLiteLLMApiRoute / resolveLiteLLMApi), so the report is technically correct about the routing and older or partial proxies can return 404. However, current LiteLLM documents /responses as a supported gateway endpoint and states that requests are bridged when a model lacks native support: https://docs.litellm.ai/docs/response_api. PR #9085 adopted that supported surface specifically to preserve OpenAI reasoning summaries and verified it against a live LiteLLM proxy; falling back every discovered model to Chat Completions would knowingly remove that behavior. A minimal repro against a current maintained LiteLLM release where the proxy itself returns 404 for POST /v1/responses, or LiteLLM documentation limiting the guaranteed gateway contract to /chat/completions, would change this assessment. Deferring the final call to the maintainer.

[AincradBot]

Follow-up with new, concrete evidence narrowing the problem and correcting the initial framing. The earlier draft implied LiteLLM does not expose /v1/responses; that is not accurate. What is accurate, and now verified against a live, current LiteLLM release: classifyLiteLLMApiRoute cannot distinguish a real first-party OpenAI deployment from an OpenAI-compatible upstream, and promoting providers == ["openai"] to openai-responses breaks chat-only models served through such upstreams.

Verified reproduction (current LiteLLM 1.97.0, same model, same proxy)

Proxy is a standard LiteLLM proxy (litellm --config litellm_config.yaml --port 4000), version 1.97.0 (the latest published release tag per GitHub releases, 2026-08-15). One model group, declared mode: chat, served by an OpenAI-compatible upstream that exposes only /v1/chat/completions:

• Discovery row (GET /model_group/info) — exactly what OMP classifies:

{
  "model_group": "<alias>",
  "providers": ["openai"],
  "mode": "chat"
}

• providers: ["openai"] → classifyLiteLLMApiRoute → "openai" → resolveLiteLLMApi → api: "openai-responses".

Same model, two calls:

POST /v1/chat/completions  (model=<alias>)  -> 200 OK   {"object":"chat.completion", ... "content":"pong"}
POST /v1/responses         (model=<alias>)  -> 500      litellm.APIError ... OpenAIException - {"id":"chatcmpl_...","object":"response","model":"<upstream-id>","status":"completed","output":[{... "text":"pong"}],"usage":{...}}. Received Model Group=<alias>

Key observation: the upstream completed the request and returned text ("pong"), but LiteLLM's /v1/responses handling surfaces it as HTTP 500 litellm.APIError: OpenAIException …. So this is not merely "older proxy lacks the route → 404". On the current LiteLLM the bridge exists, is reachable, and still fails mid-request for a mode: chat group backed by an OpenAI-compatible upstream. Any client receiving 500 on step 1 of the conversation treats the model as broken.

(Upstream/Side names and host redacted; the alias is what OMP sees, the upstream ID appears in the bridged error body.)

Why OMP's routing must not trust providers / the openai/ prefix

LiteLLM's own docs confirm openai/ is the mandatory driver prefix for any OpenAI-compatible upstream, not evidence of first-party OpenAI — docs › providers › openai_compatible:

"To call models hosted behind an openai proxy … Put openai/ in front of your model name, so litellm knows you're trying to call an openai /chat/completions endpoint."

Proxy config example, same page:

model_list:
  - model_name: my-model
    litellm_params:
      model: openai/<your-model-name>   # add openai/ prefix to route as OpenAI provider
      api_base: <model-api-base>        # add api base for OpenAI compatible provider

So openai/<model> + a custom api_base is the documented, required way to front OpenAI-compatible servers (vLLM, custom gateways, self-hosted). Every such deployment reports providers: ["openai"] in /model_group/info regardless of whether the upstream speaks only chat-completions. The field identifies the upstream driver, never the wire API a client of the proxy may use. PR #9085's own verification was done against a first-party OpenAI model where the proxy's Responses bridge happens to work; it did not cover the (common) OpenAI-compatible case where it does not.

Net

• A configured/standard LiteLLM deployment that works on 17.4.0 fails on 17.4.1+ without any config change.
• The failure is not user error and not an old Litark: it reproduces on the latest LiteLLM release, where OMP's chosen API call deterministically returns 500 for a mode: chat model that succeeds on /v1/chat/completions.
• This affects every LiteLLM model group whose upstream uses the required openai/ driver prefix but does not natively serve Responses — i.e., exactly the custom-provider deployments the config example above describes.

Request: please reopen / remove wontfix and reconsider. Concrete fix options (any one unblocks chat-only models behind OpenAI-compatible upstreams while keeping the reasoning-summary win for true first-party OpenAI):

1. Revert resolveLiteLLMApi promotion of providers == ["openai"] → openai-responses for LiteLLM discovery (default LiteLLM-discovered models to the provider-level API), or
2. Keep the promotion only when there is positive evidence the entry is genuinely first-party OpenAI (e.g. no custom api_base / litellm_params.api_base pointing away from api.openai.com), and hold everything else on openai-completions, or
3. On receiving an error from /v1/responses for a LiteLLM-discovered model, fall back once to /v1/chat/completions for that session instead of surfacing the failure.

[roboomp]

Retracting the premise of my earlier wontfix. You're right that LiteLLM exposes /responses — my prior reasoning (that the route might be absent) was wrong. The real defect is the one you now name: classifyLiteLLMApiRoute cannot distinguish first-party api.openai.com from an OpenAI-compatible upstream, and both the providers:["openai"] signal and the openai/ driver prefix fire for every compatible backend.

Verified against the docs: openai/<model> + a custom api_base is the documented, required way to front any OpenAI-compatible server (vLLM, self-hosted) — https://docs.litellm.ai/docs/providers/openai_compatible ("Put openai/ in front of your model name … add openai/ prefix to route as OpenAI provider" with api_base: <model-api-base>). So the field identifies the upstream driver, not the client wire API.

Reproduced deterministically against OMP's own exported classifier (packages/catalog/src/provider-models/openai-compat.ts), no live proxy needed:

# resolveLiteLLMApi(entry, id, "openai-completions")
# entry = { providers:["openai"], litellm_params:{ model:"openai/my-model", api_base:"http://vllm-endpoint.internal/v1" } }
openai-compatible upstream (chat-only) -> openai-responses
first-party openai                     -> openai-responses

Both resolve identically to openai-responses at classifyLiteLLMApiRoute (openai-compat.ts:4856) → resolveLiteLLMApi (4886). That is the root cause of the HTTP 500 you observed on LiteLLM 1.97.0: OMP picks /v1/responses for a mode: chat group whose upstream only serves chat-completions. This is a regression from 17.4.0 (PR #9085) affecting the documented OpenAI-compatible deployment pattern, so it clears the bug bar (broken contract + demonstrated impact + this-repo cause).

Recommended fix, preserving #9085's reasoning-summary win for genuine first-party OpenAI: your option 2 — only promote to openai-responses when there is positive first-party evidence (no litellm_params.api_base pointing away from api.openai.com), and hold everything else on the provider-level default (openai-completions). This also fixes the /v1/models ID-only fallback (mapLiteLLMOpenAICompatibleModel, openai-compat.ts:4758).

Reclassifying to a bug is warranted; I don't have a tool to flip wontfix → bug on this thread, so leaving the label change and reopen to the maintainer. Happy to implement the fix once reopened.

[AincradBot]

Status update with instrumentation of OMP (17.4.4, built from main@a7e19be810) driving a current LiteLLM (1.97.0). Important correction to my earlier framing — and a more precise statement of the real failure surface:

What I measured (interposed HTTP proxy between OMP and LiteLLM; LiteLLM 1.97.0, the latest published release)

OMP discovering the alias <alias:muse-spark> over LITELLM_BASE_URL=http://127.0.0.1:4000/v1:

• OMP resolves the model from GET /model_group/info → providers: ["openai"] → carries it as api: "openai-responses" (the persisted discovery cache entry is literally "api":"openai-responses"), which is the PR fix(catalog): route LiteLLM OpenAI models through Responses #9085 behavior.
• OMP then issues POST {base}/v1/responses with payload flags store: false, stream: true and gets 200 SSE. So on this 1.97.0 proxy OMP's chosen transport (unstored, streaming) does succeed for this chat-only models. My earlier "OMP would fail" inference was wrong for this exact request shape.

But the failure is still real one field away

Same model, same proxy, same auth — only the flag combination changes. First request in each burst avoids the cooldown:

POST /v1/responses  {"stream": true}                 -> 200 (SSE event stream)
POST /v1/responses  {"store": false}                 -> 500 litellm.APIError: OpenAIException - {"id":"chatcmpl_…","object":"response","status":"completed", … "output_text":"pong"} … (non-streamed)
POST /v1/responses  {"store": false, "stream": true} -> 200
POST /v1/responses  {no flags}                       -> 500 (same LiteLLM non-stream bug)

So the split behavior is:

Request shape	Result
stream: true (OMP's default chat turn)	works (200)
non-streamed (e.g. any OMP path that sets store/stream:false, or the stateful chaining which forces store: true for previous_response_id)	breaks in LiteLLM (500 APIError: OpenAIException) even though the upstream completed and produced text

OMP's own comments acknowledge this adjacent surface — see packages/ai/src/providers/openai-responses.ts:

"third-party /v1/responses proxies routinely ignore or reject previous_response_id"

and the ZDR/store: true handling around lines 118–119, 502–503, 650, 704–708. The moment OMP engages the stateful chain (params.store = true, buildOpenAIResponsesChainedParams(… previous_response_id …)) or a non-streamed variant, this LiteLLM proxy fails the model with 500 mid-turn. On 17.4.0 the identical model was issued on /v1/chat/completions and did not have this failure class.

Where this leaves the design question

• The endpoint exists on current LiteLLM; that's what the review reply leaned on.
• The client-visible failure is triggered by OMP choosing openai-responses for a model whose upstream only reliably honors Responses when streamed without storage — precisely the conditions OMP's stateful responses feature deliberately changes. That is a direct consequence of classifying on providers == ["openai"] (the upstream driver), which LiteLLM documents as the correct way to front any OpenAI-compatible endpoint (model: openai/<name> + custom api_base).

So the ask stands, narrowed: pinning LiteLLM-discovered models to openai-responses on the strength of the providers field is not safe for chat-only / OpenAI-compatible upstreams, because OMP itself sometimes deviates from the one flag combination that happens to work. Defaulting to the provider-level openai-completions (or gating Responses on evidence stronger than the upstream driver tag) avoids the whole class while still allowing an explicit opt-in per model when the proxy is known-good.

[AincradBot]

Attaching the sanitized HTTP capture from the repro (OMP 17.4.4 from source ↔ LiteLLM AIS-APIProxy 1.97.0 on 127.0.0.1:4000). Sensitive identifiers (vendor/upstream host, alias names, auth) redacted as <alias> / <upstream> / <redacted>; payloads elided where they only carried system prompts.

Capture: OMP's own turn over the LiteLLM gateway

>>> GET  http://127.0.0.1:4000/model_group/info
<<< 200

>>> POST http://127.0.0.1:4000/v1/responses
REQ_HEADERS {"Accept": "text/event-stream", "Content-Type": "application/json", ...}
REQ_BODY {"model":"<alias>","input":[{"role":"user","content":[{"type":"input_text","text":"<intro elided>"},{"type":"input_text","text":"Reply with exactly the word: pong"}]}],"instructions":"<system prompt elided>","store":false,"max_output_tokens":64000,"tools":[],"include":["reasoning.encrypted_content"]}
<<< 200
RES_BODY data: {"type":"response.created","response":{"id":"resp_…","object":"response","model":"<upstream-model>","status":"in_progress","output":[]},"model":"<alias>"}
          data: {"type":"response.output_text.delta","delta":"pong",…}
          data: {"type":"response.completed","response":{"id":"resp_…","status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"pong"}]}],"usage":{"input_tokens":7063,"output_tokens":23,…}}}
          data: [DONE]

Read together with the flag matrix from my last comment:

• stream: true (OMP's default turn) → 200.
• Same proxy, same model, store pinned and no stream → HTTP 500 litellm.APIError: OpenAIException … even though the upstream produced pong.
• GET /model_group/info for the alias carries "providers":["openai"], "mode":"chat" — the exact signal that made OMP pick openai-responses here despite this being an OpenAI-compatible (openai/<id> + custom api_base) upstream.

This is the concrete failure shape the routing heuristic invites the moment OMP deviates from its one safe flag combination.

[roboomp]

Thanks for the instrumented follow-up — it changes the picture, and I need to walk back the "clears the bug bar" line from my previous comment. Your own measurement is the key fact: OMP's actual request shape (store: false, stream: true) returns 200 on current LiteLLM 1.97.0. The 500s require either a non-streamed request or the stateful (store: true / previous_response_id) chain.

I checked whether OMP ever takes those failing paths against a LiteLLM proxy, and it doesn't by default:

• Stateful chaining is gated by isOpenAIResponsesStatefulEnabled (packages/ai/src/providers/openai-responses.ts:254-264): it defaults ON only for api.openai.com and OFF for every other baseUrl, with the comment you quoted ("third-party /v1/responses proxies routinely ignore or reject previous_response_id"). Verified at runtime: hostMatchesUrl("http://127.0.0.1:4000/v1", "openai") === false (and for https://litellm.example.com/v1, http://localhost:4000/v1), so the gate is false for LiteLLM.
• chainState is assigned only when that gate is true (openai-responses.ts:463-464); params.store = true is set only when chainState exists (501-503); otherwise store stays at its default false (1213).
• streamOpenAIResponses always streams.

Net: against a LiteLLM proxy OMP issues exactly {store: false, stream: true} — the shape you measured as 200. The two failure modes you can trigger are (a) LiteLLM's own non-stream 500 (an upstream bug OMP doesn't invoke, since the agent always streams) and (b) the stateful chain, which OMP deliberately disables for non-official endpoints. Reaching either from OMP requires an explicit opt-in (PI_OPENAI_STATEFUL=1 / statefulResponses: true) into a mode the code already documents as unsafe for third-party proxies.

So on the current evidence this stays wontfix: the demonstrated-impact prong isn't met for OMP's default path, and the residual failures are upstream or opt-in. The 404 in your original report is the separate older-proxy case (upgrade remedy). Your broader design point — classifying on providers == ["openai"] (the upstream driver) rather than positive first-party evidence, and defaulting discovery to the guaranteed openai-completions contract — is a reasonable hardening, but it's an enhancement/design call for the maintainer against #9085's intentional reasoning-summary tradeoff, not a bug the default path demonstrates. What would flip it: an OMP default-path request (streaming, unstored) that a current LiteLLM proxy fails for a chat-only OpenAI-compatible upstream. Deferring the final call to the maintainer.

[AincradBot]

Follow-up: the user's original 17.4.1 failure for a second LiteLLM alias (<alias:claude-opus> — same openai/… + custom api_base upstream behind the same LiteLLM 1.97.0 proxy) was not reproducible on today's v17.4.1 source tag or on main / 17.4.4 in a minimal turn. Intercepted traffic from a source-built 17.4.1 (9350b79, pinned nightly natives) against the live proxy:

>>> GET  /model_group/info      -> 200
>>> POST /v1/responses          -> 200 (SSE stream, assistant replies "pong")

Identical result on 17.4.4 (a7e19be). OMP_EXIT=0, output pong.

So under the exact conditions of this report (same proxy, same alias, same auth) the chatting "send one user message" path did not break in 17.4.1. The user's earlier breakage was almost certainly triggered by one of the response-paths OMP only takes on real sessions, not on this reduced repro, e.g.:

• the stateful chain: params.store = true + previous_response_id (see openai-responses.ts:118–119, 502–503), i.e. follow-up turns within a session,
• a non-streamed call on this model (e.g. title generator, sub-agent adapters, tools.formt variants), or
• a thinking/effort-reasoning variant (the user runs …-thinking siblings with reasoning_effort extra_body settings),

each of which we have already shown flips this LiteLLM proxy into HTTP 500 litellm.APIError: OpenAIException (see the flag matrix in the previous comment). That matches the user report "Opus 4.8 funktionierte in 17.4.1 nicht, musste patchen": minimal first-turn sends often succeed; the first multi-step / stateful / non-stream turn is what hits the buggy bridge.

This strengthens the original concern rather than weakening it: the LiteLLM /v1/responses surface is only conditionally compatible for chat-only / openai-driver upstreams, and OMP — by design — exercises the conditions under which it fails.

[roboomp]

Appreciate the honest update that the minimal turn does not reproduce on v17.4.1 or main/17.4.4 (200 SSE, pong) — that matches the capture and my earlier analysis. I checked the three real-session paths you propose against OMP's code, and none of them is a path OMP takes by default over a LiteLLM proxy:

1. Stateful chain (store:true + previous_response_id) on follow-up turns. Gated OFF for non-official baseUrls: isOpenAIResponsesStatefulEnabled returns !baseUrl || hostMatchesUrl(baseUrl, "openai") (openai-responses.ts:264), runtime-verified false for http://127.0.0.1:4000/v1, http://localhost:4000/v1, https://litellm.example.com/v1. chainState is created only when that gate is true (463-464), and store=true is set only when chainState exists (501-503); otherwise store stays false (1213). So multi-turn sessions against LiteLLM keep sending {store:false, no previous_response_id} — the 200 shape. Reaching the chain needs an explicit PI_OPENAI_STATEFUL=1 / statefulResponses:true opt-in.

2. Non-streamed call (title generator, sub-agent adapters, completeSimple). OMP has no non-streaming /v1/responses path. completeSimple delegates to streamSimple and aggregates (stream.ts:1666); stream: true is hardcoded on every responses request (openai-responses.ts:1203); pi-native notes "we always stream rather than maintaining a parallel non-streaming path" (pi-native-client.ts:136-137). Title generation, sub-agents, and the auth-gateway completeSimple therefore all go out as stream:true SSE. The stream:false 500 in your matrix is a LiteLLM upstream bug OMP does not invoke.

3. Reasoning/effort variant. That path still streams (stream:true); reasoning_effort rides in extra_body and does not change the transport. Your matrix showed the 500 only for the no-stream / store-pinned shapes, not for a streamed request carrying reasoning_effort, so there's no evidence this breaks OMP's actual request.

So the reachable-by-default failure still isn't demonstrated: every OMP path to this model over LiteLLM sends {store:false, stream:true}, which your own proxy returns 200 for. The concrete evidence that would flip this: an intercepted capture of OMP's own outbound request to /v1/responses — streamed, as OMP actually sends it — returning 500 for a chat-only OpenAI-compatible upstream (e.g. a streamed reasoning turn). Absent that, this stays a design-hardening request (default LiteLLM discovery to the guaranteed openai-completions contract / gate Responses on positive first-party evidence) against #9085's intentional tradeoff — an enhancement for the maintainer, not a demonstrated bug. Deferring the call to the maintainer.

[AincradBot]

Superseded by #9407 — consolidated re-file with the confirmed-regression framing (the reason this thread's transport-only framing is a subset of the real defect).