[Feature]: Model Failover Policy

[Thenujan-Nagaratnam]

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI)

Suggested Feature

An application built on top of an LLM proxy API has no way to stay up when the specific model or provider it's calling degrades or goes down — a 500-class response from the primary model is simply handed back to the app as-is. Today the app developer has to write and maintain their own retry-and-fallback logic client-side: catch the failure, pick a fallback model, potentially re-authenticate and re-shape the request for a completely different provider's API, and retry — for every app that calls the gateway.

We need a policy for this and automatically fallback to the configured models when sth goes wrong with a model.

Related Issues

No response

Steps to Verify

• Design Document — A detailed design document has been created and reviewed, covering architecture, data flow, and edge cases.
• Design Mail — A design summary email has been sent to relevant stakeholders for awareness and feedback.
• Code Review — All code changes have been peer-reviewed and approved according to the project's review standards.
• Testing Complete — Adequate unit, integration, and/or end-to-end tests have been written and are passing.
• Documentation Review — User-facing and/or developer documentation has been updated to reflect the new feature and reviewed.
• Feature Complete — The feature is fully implemented, all checklist items above are done, and it is ready for release.

[Thenujan-Nagaratnam]

Retry Mechanism: Internal Discussion Summary

This document summarizes our architectural discussion regarding the implementation of a retry mechanism to support OAuth2 Token Refresh and LLM Model Failover.

1. Primary Use Cases & Requirements

The retry mechanism must handle two distinct scenarios with varying levels of complexity:

• OAuth2 Policy (Token Refresh):
  • Requires catching authentication failures (e.g., 401s), fetching a new token, modifying the request headers, and re-invoking the endpoint.
• Model Failover (Cross-Provider Routing):
  • Requires falling back to alternative LLM providers (e.g., OpenAI → Anthropic) when the primary model fails.
  • Required Modifications during Retry:
    • Endpoint & Path: Routing to an entirely different provider URL (e.g., /chat/completions to /messages).
    • Body Transformation: Translating the payload to match the fallback provider’s expected schema.
    • Authentication: Swapping credentials to use the correct auth method for the fallback provider.
    • Headers: Adding or modifying headers as required by the new target (future-proofing).

2. Envoy Native Capabilities Analysis

We analyzed Envoy’s native RetryPolicy combined with upstream ext_proc (which sits between the router filter and the backend) and Aggregate Clusters.

• Upstream ext_proc: Can handle modifications to headers, body, and path. Limitation: Cannot modify the target endpoint (retries hit the same upstream).
• Aggregate Clusters: Allows defining an ordered list of fallback endpoints per retry attempt, solving the dynamic endpoint routing issue.
• Outstanding Gaps: While combining these features routes the request, native Envoy does not easily solve cross-provider authentication swapping and body transformations.

3. Approaches Considered

Approach	Status	Description & Assessment
1. Upstream ext_proc Retry	🛑 Blocked	Handle retries natively via Envoy's upstream hook. Drawbacks: The upstream ext_proc is still alpha and unfit for GA. Furthermore, standardizing transformation and authentication logic for multiple providers within this hook is complex and diverges from our existing policy logic.
2. Policy-Engine Re-route (Response Path)	🟡 Have to evaluate	On a failure (e.g., 401) in the response path, the policy engine re-invokes the route. Concept: If an API has an OAuth2 policy, create a parallel "bare" route (no policy chain). On 401, the engine refreshes the token and calls the bare route. Pros: Credential material is already accessible from the policy engine. Cons: Generalizing this to cross-provider failover (where the retry needs completely different auth/payloads, not just a refresh) remains an open challenge.
3. Lua Filters	❌ Ruled Out	Reasoning: The team is discussing about deprecating Lua filters in the gateway.

This is the branch I used for the Approach 01's POC: https://github.com/Thenujan-Nagaratnam/api-platform/tree/decoupled-retry-source