feat(llm): add multi-replica worker routing

Signed-off-by: Mike Camp <mcamp@nvidia.com>

Author: mikeyrcamp

.github/workflows/image-push-manual.yml

@@ -167,6 +167,7 @@ jobs:
             exit 1
           fi
           echo "discovered: ${indexes[*]}"
+          declare -A seen_repos=()
           for tgt in "${indexes[@]}"; do
             name="${tgt##*:}"; name="${name%_index}"
             # Two naming conventions exist in the tree and they mean different
@@ -186,6 +187,12 @@ jobs:
               *-image) repo="${name%-image}" ;;
               *)       sub="$(printf '%s' "$name" | tr '_' '-')"; repo="${svc}-${sub}" ;;
             esac
+            if [ -n "${seen_repos[$repo]:-}" ]; then
+              echo "ERROR: ${tgt} and ${seen_repos[$repo]} both map to image repository ${repo}" >&2
+              echo "Give each oci_image_index target a distinct image name before publishing." >&2
+              exit 1
+            fi
+            seen_repos["$repo"]="$tgt"
             dest="${REGISTRY}/${repo}"
             echo "[push] ${tgt} -> ${dest}:${TAG} (+ latest-dispatch)"
             mkdir -p ci-ghcr

deploy/helm/gateway-routes/README.md

@@ -4,17 +4,26 @@ This repository contains the Helm chart for deploying NVCF ingress routes via th
 
 ## Overview
 
-The chart deploys `HTTPRoute`, `TCPRoute`, and `ReferenceGrant` resources that attach to an existing Gateway provisioned separately by the cluster operator (e.g. Envoy Gateway, Istio, Traefik, Kong). It also includes optional `PodMonitor` resources for scraping Envoy Gateway proxy metrics with Prometheus.
-
-The chart deploys routing configuration only. It does not include any container images. Backend services referenced by the routes (`api`, `nvct-api`, `api-keys`, `invocation`, `llm-api-gateway`, `vanity-gateway`, `reval`, `sis`, `grpc`, `nats`) must already be deployed separately.
+The chart deploys `HTTPRoute`, `GRPCRoute`, `TCPRoute`, `UDPRoute`, and
+`ReferenceGrant` resources that attach to an existing Gateway provisioned
+separately by the cluster operator, such as Envoy Gateway, Istio, Traefik, or
+Kong. It also includes optional `PodMonitor` resources for scraping Envoy
+Gateway proxy metrics with Prometheus.
+
+The chart deploys routing configuration only. It does not include any
+container images. Backend services referenced by the routes (`api`,
+`nvct-api`, `api-keys`, `invocation`, `llm-api-gateway`,
+`llm-request-router-backend-router`, `vanity-gateway`, `reval`, `sis`, `grpc`,
+`nats`) must already be deployed separately.
 
 ## Prerequisites
 
 - Kubernetes cluster
 - Helm 3.x
 - `kubectl`
 - A Gateway API compatible controller installed in the cluster
-- An existing `Gateway` resource with an HTTP listener (and TCP listeners if the gRPC or NATS routes are enabled)
+- Existing `Gateway` resources with the listeners required by each enabled route
+- A Gateway controller with `UDPRoute` support when LLM worker routing is enabled
 - The backend services that the routes target, deployed in their respective namespaces
 
 ## Getting Started
@@ -57,6 +66,8 @@ Important settings to review before deployment:
 - `nvcfGatewayRoutes.gateways.shared.*` for the HTTP Gateway name, namespace, and listener
 - `nvcfGatewayRoutes.gateways.grpc.*` for the TCP Gateway name, namespace, and listener
 - `nvcfGatewayRoutes.gateways.nats.*` for the NATS TCP Gateway name, namespace, and listener
+- `nvcfGatewayRoutes.gateways.llmGrpc.*` for the LLM worker gRPC TCP listener
+- `nvcfGatewayRoutes.gateways.llmQuic.*` for the LLM reverse-tunnel UDP listener
 - `nvcfGatewayRoutes.routes.<route>.enabled` to toggle individual routes
 - `nvcfGatewayRoutes.routes.nvcfApi.grpc.enabled` and
   `nvcfGatewayRoutes.routes.nvctApi.grpc.enabled` to expose API gRPC routes
@@ -87,6 +98,7 @@ Enabled `HTTPRoute` entries must not share a resolved hostname because each `HTT
 | `grpc` | TCPRoute | Not rendered | `grpc.nvcf:10081` |
 | `grpcWorker` | TCPRoute (disabled by default) | Not rendered | `grpc.nvcf:10086` |
 | `nats` | TCPRoute (disabled by default) | Not rendered | `nats.nats-system:4222` |
+| `llmWorker` | TCPRoute and UDPRoute (disabled by default) | Not rendered | `llm-request-router-backend-router.nvcf:50071/TCP,50072/UDP` |
 
 Cross-namespace routing is supported via `ReferenceGrant` resources rendered into each backend namespace.
 
@@ -97,3 +109,6 @@ Cross-namespace routing is supported via `ReferenceGrant` resources rendered int
 - The `grpc` TCPRoute does not enforce HTTP hostname matching at the Gateway layer. Configure DNS or TCP load balancer routing outside this chart.
 - The `grpcWorker` TCPRoute is beta support for split or multi-cluster gRPC worker callbacks. It carries HTTP/1 CONNECT callback traffic only. Enable it only when the control-plane grpc-proxy runs one replica with HPA disabled. Multi-replica grpc-proxy requires pod-specific callback routing and is not supported by this shared TCPRoute.
 - Enabling the `nats` route requires a reachable TCP listener for NATS on the referenced Gateway. The HTTP Gateway address does not imply NATS reachability unless that same Gateway also has the NATS TCP listener configured.
+- The `llmWorker` routes target Stargate's authority/SNI-aware backend router.
+  Keep the TCP and UDP Gateways separate when the infrastructure requires
+  separate load balancers for each protocol.

deploy/helm/gateway-routes/chart/templates/referencegrant-llm-worker.yaml

@@ -0,0 +1,24 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: allow-llm-worker-routes
+  namespace: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+spec:
+  from:
+    - group: gateway.networking.k8s.io
+      kind: TCPRoute
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+    - group: gateway.networking.k8s.io
+      kind: UDPRoute
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.namespace }}
+  to:
+    - group: ""
+      kind: Service
+      name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.name }}
+{{- end }}

deploy/helm/gateway-routes/chart/templates/tcproute-llm-worker.yaml

@@ -0,0 +1,27 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+apiVersion: gateway.networking.k8s.io/v1alpha2
+kind: TCPRoute
+metadata:
+  name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-grpc
+  namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+    app.kubernetes.io/component: llm-worker-grpc-route
+  {{- with .Values.nvcfGatewayRoutes.routes.llmWorker.routeAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  parentRefs:
+    - name: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.name }}
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+      sectionName: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.listenerName }}
+  rules:
+    - backendRefs:
+        - name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.name }}
+          namespace: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.namespace }}
+          port: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.grpcPort }}
+{{- end }}

deploy/helm/gateway-routes/chart/templates/udproute-llm-worker.yaml

@@ -0,0 +1,27 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+apiVersion: gateway.networking.k8s.io/v1alpha2
+kind: UDPRoute
+metadata:
+  name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-quic
+  namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+    app.kubernetes.io/component: llm-worker-quic-route
+  {{- with .Values.nvcfGatewayRoutes.routes.llmWorker.routeAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  parentRefs:
+    - name: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.name }}
+      namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.namespace }}
+      sectionName: {{ .Values.nvcfGatewayRoutes.gateways.llmQuic.listenerName }}
+  rules:
+    - backendRefs:
+        - name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.name }}
+          namespace: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.namespace }}
+          port: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.backend.quicPort }}
+{{- end }}

deploy/helm/gateway-routes/chart/values.yaml

@@ -51,6 +51,17 @@ nvcfGatewayRoutes:
       namespace: gateway
       # Listener (sectionName) on the Gateway to attach the route to
       listenerName: nats
+    # TCP Gateway for LLM worker registration and Stargate watches.
+    llmGrpc:
+      name: llm-grpc-gateway
+      namespace: gateway
+      listenerName: llm-grpc
+    # UDP Gateway for LLM reverse tunnels. This may be a separate load
+    # balancer from the TCP Gateway.
+    llmQuic:
+      name: llm-quic-gateway
+      namespace: gateway
+      listenerName: llm-quic
 
   # HTTPRoute configurations
   routes:
@@ -239,6 +250,19 @@ nvcfGatewayRoutes:
         namespace: nats-system
         port: 4222
       routeAnnotations: {}
+
+    # Backend-facing LLM registration/watch and reverse-tunnel routes.
+    # The backend router selects the correct Stargate pod by gRPC authority
+    # and QUIC SNI, so this route supports multi-replica request routers.
+    llmWorker:
+      enabled: false
+      name: llm-worker
+      backend:
+        name: llm-request-router-backend-router
+        namespace: nvcf
+        grpcPort: 50071
+        quicPort: 50072
+      routeAnnotations: {}
 
     # NVCF Worker Container needs to be able to fetch secrets for function pods
     # In split/multi cluster planes, ess resides in control-plane and requires

deploy/helm/gateway-routes/scripts/check-llm-worker-routes.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+chart_dir="${script_dir}/../chart"
+rendered="$(mktemp)"
+disabled="$(mktemp)"
+trap 'rm -f "$rendered" "$disabled"' EXIT
+
+helm template nvcf-gateway-routes "$chart_dir" \
+  --namespace gateway \
+  --set nvcfGatewayRoutes.routes.llmWorker.enabled=true \
+  --set nvcfGatewayRoutes.gateways.llmGrpc.name=llm-grpc-gateway \
+  --set nvcfGatewayRoutes.gateways.llmGrpc.namespace=gateway \
+  --set nvcfGatewayRoutes.gateways.llmQuic.name=llm-quic-gateway \
+  --set nvcfGatewayRoutes.gateways.llmQuic.namespace=gateway \
+  >"$rendered"
+
+assert_contains() {
+  local pattern="$1"
+  local message="$2"
+  if ! grep -Fq -- "$pattern" "$rendered"; then
+    echo "FAIL: ${message}" >&2
+    exit 1
+  fi
+}
+
+assert_contains "kind: TCPRoute" \
+  "LLM worker routing must expose gRPC registration over TCP"
+assert_contains "kind: UDPRoute" \
+  "LLM worker routing must expose reverse tunnels over UDP"
+assert_contains "name: llm-request-router-backend-router" \
+  "LLM worker routes must target the authority/SNI-aware backend router"
+assert_contains "name: allow-llm-worker-routes" \
+  "ReferenceGrant must permit cross-namespace LLM worker routes"
+assert_contains "sectionName: llm-grpc" \
+  "TCPRoute must attach to the configured LLM gRPC listener"
+assert_contains "sectionName: llm-quic" \
+  "UDPRoute must attach to the configured LLM QUIC listener"
+
+helm template nvcf-gateway-routes "$chart_dir" \
+  --namespace gateway \
+  --set nvcfGatewayRoutes.routes.llmWorker.enabled=false \
+  >"$disabled"
+
+if grep -Eq '^  name: (llm-worker-(grpc|quic)|allow-llm-worker-routes)$' "$disabled"; then
+  echo "FAIL: disabled LLM worker routing must not render route resources" >&2
+  exit 1
+fi
+
+echo "PASS: LLM worker Gateway routes render correctly"

deploy/helm/llm-request-router/README.md

@@ -4,7 +4,17 @@ This repository contains the Helm chart for deploying the NVCF LLM Request Route
 
 ## Overview
 
-The chart packages the LLM Request Router StatefulSet with HTTP and gRPC services, a metrics endpoint, and a headless service for multi-instance DNS discovery. A Vault Agent sidecar is configured to fetch a service token from a Vault or OpenBao backend; the application reads `nvcfApiToken` from `/vault/secrets/secrets.json` and attaches it as a Bearer token to outgoing worker authentication gRPC calls.
+The chart packages the LLM Request Router StatefulSet with HTTP and gRPC
+services, a metrics endpoint, and a headless service for multi-instance DNS
+discovery. It can also deploy the Stargate Kubernetes backend router for
+worker gRPC registration and reverse QUIC tunnels through a shared Gateway or
+load balancer. The backend router selects the correct Stargate pod from gRPC
+authority and QUIC SNI.
+
+A Vault Agent sidecar is configured to fetch a service token from a Vault or
+OpenBao backend. The application reads `nvcfApiToken` from
+`/vault/secrets/secrets.json` and attaches it as a Bearer token to outgoing
+worker authentication gRPC calls.
 
 The default chart values do not set the required image registry and repository. They must be supplied through an additional values file at install time, and access to those images must be arranged separately.
 
@@ -70,6 +80,7 @@ Important settings to review before deployment:
 - `llmRequestRouter.imagePullSecrets` for private registry access
 - `llmRequestRouter.replicaCount`, resource requests, and limits for your environment
 - `llmRequestRouter.service.*` for HTTP, gRPC, metrics, and headless service ports
+- `llmRequestRouter.backendRouter.*` for multi-replica worker gRPC and reverse-tunnel routing
 - `llmRequestRouter.metrics.enabled` to expose the metrics port on the Service (default: `false`)
 - `llmRequestRouter.metrics.serviceMonitor.enabled` to create a Prometheus `ServiceMonitor` (requires `metrics.enabled`)
 - `llmRequestRouter.certificate.*` to let cert-manager issue the Stargate QUIC server certificate
@@ -80,6 +91,51 @@ Important settings to review before deployment:
 
 The default values include development-oriented placeholders. Override them before using the chart in any shared or production environment.
 
+## Backend Worker Routing
+
+Enable `llmRequestRouter.backendRouter.enabled` when workers reach a
+multi-replica request router through a shared endpoint. Set both pylon dial
+addresses to the external endpoints that workers can resolve:
+
+```yaml
+llmRequestRouter:
+  backendRouter:
+    enabled: true
+    image:
+      tag: <published-stargate-version>
+    pylonGrpcDialAddress: llm-router.example.com:443
+    pylonReverseTunnelDialAddress: llm-router.example.com:8080
+```
+
+The chart uses the main Stargate image for both workloads. The image must
+contain `/usr/local/bin/stargate-k8s-router`. Set
+`llmRequestRouter.backendRouter.image.tag` to an image version that contains
+that binary. The chart requires this explicit pin when backend routing is
+enabled.
+
+The backend router watches EndpointSlices. The chart creates a namespaced Role
+and RoleBinding when `llmRequestRouter.rbac.create=true`. If the chart does not
+create the ServiceAccount, set `llmRequestRouter.serviceAccount.name`
+explicitly so the RoleBinding does not grant access to the namespace default
+ServiceAccount. When `rbac.create=false`, grant `get`, `list`, and `watch` on
+`discovery.k8s.io/endpointslices` to the selected ServiceAccount outside this
+chart.
+
+Route TCP port `50071` and UDP port `50072` to the
+`llm-request-router-backend-router` Service. The NVCF gateway-routes chart can
+create the matching `TCPRoute`, `UDPRoute`, and `ReferenceGrant` resources.
+The Gateway implementation must support Gateway API `UDPRoute`.
+
+When QUIC verification is enabled, the mounted certificate must cover the
+worker-facing reverse-tunnel hostname and the per-pod hostname template. The
+default template is
+`{pod_name}.llm-request-router-headless.<namespace>.svc.cluster.local`.
+
+Stargate and the backend router read the TLS certificate and key only during
+process startup. After cert-manager or another issuer renews the Secret,
+restart both workloads or configure a Secret reloader that triggers their
+rollouts.
+
 ## Load Balancer Configuration
 
 The chart can pass a Stargate load-balancer config in either of two ways: