Skip to content

[Feature]: Multi-provider LLM proxies and selectable inbound interface as an experience in AI Workspace #3368

Description

@veejask-41

Please select the area the issue is related to

Gateway, Platform API, AI Workspace

Please select the aspect the issue is related to

Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI), Aspect/UI (Frontend layouts, components, styling), Aspect/UX (User experience, flows, usability, clarity), Aspect/AI (AI/LLM integration, MCP, AI readiness)

Suggested Feature

Problem

The gateway supports routing one LLM proxy to multiple providers, with per-provider transformer policies converting between wire formats. None of this is reachable from AI Workspace.

Two gaps today:

  1. A proxy connects to exactly one provider. additionalProviders cannot be expressed in the UI.
  2. The inbound interface is silently inherited. A proxy's request format is copied from its primary provider at creation and never presented as a choice, so "expose an OpenAI-compatible endpoint in front of a Mistral provider" is not expressible.

The underlying capability already exists end to end in the gateway (additionalProviders, llm-header-router, five openai-to-* transformer policies) and is partially wired through Platform API.

Proposed solution

Introduce inbound interface as an explicit proxy setting, and allow attaching multiple providers to a proxy.

Core model

The inbound interface is the request format client applications send. It defaults to the primary provider's template and is identified by a template handle (openai, anthropic, gemini, mistralai, …). Its only job is resolving which transformer each attached provider needs:

Condition Result
provider template == interface No transformer needed
provider template != interface Requires policy {interface}-to-{template}-transformer
that policy is absent Provider can be attached but proxy cannot be deployed without required transformer

This one rule covers both gaps. Multi-provider fan-out is many providers reconciled to one interface; "OpenAI interface on a Mistral proxy" is one provider whose template differs from the interface.

Related Issues

No response

Steps to Verify

  • Design Document — A detailed design document has been created and reviewed, covering architecture, data flow, and edge cases.
  • Design Mail — A design summary email has been sent to relevant stakeholders for awareness and feedback.
  • Code Review — All code changes have been peer-reviewed and approved according to the project's review standards.
  • Testing Complete — Adequate unit, integration, and/or end-to-end tests have been written and are passing.
  • Documentation Review — User-facing and/or developer documentation has been updated to reflect the new feature and reviewed.
  • Feature Complete — The feature is fully implemented, all checklist items above are done, and it is ready for release.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Area/AIGatewayAI Gateway runtime/control plane related issuesArea/AIPoliciesIssues related to policies,guardrails in AI GatewayArea/AIWorkspaceIssues related to AI WorkspaceAspect/AIAI/LLM integration, MCP, AI readinessAspect/APIAPI definitions, contracts, OpenAPI, interfacesAspect/UIFrontend, visual layouts, components, stylingAspect/UXUser experience, flows, usability, clarityPriority/HighSeverity/MajorType/New Feature

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions