feat: introduce native model runtime contract by Paramveersingh-S · Pull Request #2530 · vllm-project/semantic-router

Paramveersingh-S · 2026-07-14T12:07:59Z

Overview

This is Phase 1 of splitting PR #2458 into smaller, staged, and behavior-preserving pull requests to address Issue #2396.

As requested in previous reviews, this PR focuses strictly on introducing the backend-neutral native runtime contracts and taxonomy. It does not touch any existing call sites or Rust logic, ensuring that the legacy candle-binding singletons and existing runtime behaviors are 100% preserved.

Key Additions

Taxonomy Lockdown (taxonomy.go): Established strict Backend, Capability, Family, ArtifactFormat, and Modality constants to eliminate string duplication and prevent future taxonomy drift.
Native Contracts (adapter.go, request.go): Defined the BackendAdapter interface that future backend adapters (Candle, ONNX, OpenVINO) will implement. Introduced CapabilitySet, LoadRequest, and InferenceRequest to formalize hardware and feature support.
Thread-Safe Registry (registry.go): Introduced the central native.Registry component to track loaded model bindings securely.
ABI Tests (ffi_layout_test.go): Added stub test files for catching layout/taxonomy drifts at the CGO boundary.

This PR only adds the new foundational types to src/semantic-router/pkg/modelruntime/native. Because no existing init() side effects, Rust code, or CGO calls were modified, this guarantees that we bypass the Rust LazyLock and Go NUL-byte CI failures encountered previously.

Fixes #2396 (part of 2396)

netlify · 2026-07-14T12:08:04Z

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	`ffebe06`
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/6a57a9bf63c9df0008a99805
😎 Deploy Preview	https://deploy-preview-2530--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

github-actions · 2026-07-14T12:08:20Z

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository:

📁 `src/semantic-router`

Owners: @FAUST-BENCHOU, @shraderdm, @drivebyer, @ramkrishs, @WUKUNTAI-0211, @AayushSaini101, @siloteemu
Files changed:

src/semantic-router/pkg/modelruntime/native/adapter.go
src/semantic-router/pkg/modelruntime/native/registry.go
src/semantic-router/pkg/modelruntime/native/registry_test.go
src/semantic-router/pkg/modelruntime/native/request.go
src/semantic-router/pkg/modelruntime/native/taxonomy.go

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

github-actions · 2026-07-14T12:09:01Z

✅ Supply Chain Security Report — All Clear

Scanner	Status	Findings
AST Codebase Scan (Py, Go, JS/TS, Rust)	✅	29 finding(s) — MEDIUM: 22 · LOW: 7
AST PR Diff Scan	✅	No issues detected
Regex Fallback Scan	✅	No issues detected

Scanned at 2026-07-15T15:44:16.150Z · View full workflow logs

Signed-off-by: Param <param15.veer.singh@gmail.com>

Xunzhuo

Thanks for splitting the original refactor. Keeping Phase 1 additive is the right boundary, and go test -race ./pkg/modelruntime/native passes on this head. I am requesting changes because the foundational contract still cannot support a backend-neutral request path, and downstream phases already expose the gaps.

Please fix this phase before rebasing the later phases:

Use a module-prefixed title such as [Router] ....
Replace Fixes #2396 with Part of #2396; Phase 1 does not satisfy the roadmap issue and must not close it when merged.
Make #2532 base on this branch, #2533 on #2532, and #2534 on #2533 (or land/rebase them sequentially). Today all four target main and contain independently recreated Phase 1 commits, so their diffs overlap and cannot be merged as a clean stack.

The inline comments cover the contract, normalized model-info shape, registry semantics, and tests. Once those are addressed, this additive first slice can be reviewed independently of the backend migrations.

Xunzhuo · 2026-07-15T03:10:54Z

+type BackendAdapter interface {
+	Name() Backend
+	Capabilities() CapabilitySet
+	LoadModel(ctx context.Context, req LoadRequest) (ModelHandle, error)


[P1] The neutral contract has no inference operation. InferenceRequest is declared, but neither BackendAdapter nor ModelHandle can execute it and there is no response type. A migrated caller would still have to import a concrete binding after LoadModel, defeating the issue's core goal that request paths ask for a capability/model ref rather than a backend package. Please put inference on the handle or adapter (with a handle argument), and use typed capability-specific request/response shapes instead of an unconsumed map[string]interface{} bag.

Xunzhuo · 2026-07-15T03:10:54Z

+	IsLoaded          bool
+	MaxSequenceLength int
+	DefaultDimension  int
+}


[P1] This is not yet the normalized model-info contract required by #2396 or consumed by Phase 4. It omits artifact format, modality, dimensions/layers as requested-vs-runtime metadata, provider/device, version, runtime feature flags/unsupported reasons, and registry metadata; it also allows only one capability. If this foundational type lands now, the API migration will either lose existing information or require an immediate breaking expansion. Please define the complete normalized discovery shape in Phase 1 and add round-trip/discovery tests before call sites depend on it.

Xunzhuo · 2026-07-15T03:10:54Z

+	r.mu.RLock()
+	defer r.mu.RUnlock()
+	list := make([]BackendAdapter, 0, len(r.adapters))
+	for _, adapter := range r.adapters {


[P1] Iterating a Go map makes List() nondeterministic. Phase 4 emits /models in exactly this order, so identical processes can return different response ordering and order-sensitive tests/clients will flake. Return adapters in a stable backend order (or sort the collected keys) and assert it with multiple registrations.

Xunzhuo · 2026-07-15T03:10:54Z

+func (r *AdapterRegistry) Register(adapter BackendAdapter) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.adapters[adapter.Name()] = adapter


[P2] Registration has no invalid/duplicate contract: a typed-nil adapter can panic at Name(), and a second adapter silently replaces the first backend. Since registration becomes the composition boundary, please return an error for nil/empty names and duplicate backends (or document and test an intentional replacement API separately). The current concurrency smoke test deliberately avoids the nil case but does not validate registry composition semantics.

Xunzhuo · 2026-07-15T03:10:55Z

+	"testing"
+)
+
+// In a real scenario, this would import C and use unsafe.Sizeof to verify structs against C headers.


[P2] This file is explicitly a placeholder and does not test FFI layout or discriminants, so it cannot be presented as the ABI drift coverage requested by #2396. Either add real Go/Rust/C boundary checks (for example generated discriminants plus unsafe.Sizeof/offset assertions against the exported C structs) or rename this to a registry/taxonomy unit test and leave ABI coverage to the Rust-binding phase. A test that only compares a Go string constant to a literal will not catch binding drift.

Signed-off-by: Param <param15.veer.singh@gmail.com>

Paramveersingh-S · 2026-07-15T15:39:33Z

Thanks for the detailed review! I've updated the PR title and description as requested. I've also pushed a new commit that addresses all your points:

[P1] Inference Contract: Replaced the map bag with strongly-typed request/response interfaces (like EmbeddingRequest) and added the Inference method to BackendAdapter.
[P1] Model-Info Shape: Expanded ModelInfo into the complete discovery shape required by Modularize native router model bindings and runtime capability contracts #2396 (added dimensions/layers, format, modality, feature flags, etc.).
[P1] Registry Determinism: List() now returns adapters in alphabetical order by backend name for stable /models responses.
[P2] Registration Rules: Register() now strictly returns an error for nil adapters, empty names, or duplicate backends.
[P2] ABI Tests: Removed the placeholder ffi_layout_test.go and replaced it with registry_test.go to test taxonomy, duplicate/nil errors, and ordering. Actual FFI boundary tests will be added in the Rust-binding phases.

The base contracts are fully updated and tested. I'll make sure to sequentially rebase the other phases

Paramveersingh-S requested review from Xunzhuo and rootfs as code owners July 14, 2026 12:08

github-actions Bot assigned AayushSaini101, drivebyer, FAUST-BENCHOU, ramkrishs, shraderdm and WUKUNTAI-0211 Jul 14, 2026

Paramveersingh-S added 3 commits July 14, 2026 18:56

feat: introduce native model runtime contract (Phase 1)

2216c2b

Signed-off-by: Param <param15.veer.singh@gmail.com>

fix: resolve nil pointer panic in tests and fix trailing whitespace

e512fe9

Signed-off-by: Param <param15.veer.singh@gmail.com>

fix: add missing context import

023ef3d

Signed-off-by: Param <param15.veer.singh@gmail.com>

Paramveersingh-S force-pushed the feature/native-router-phase1 branch from 595306e to 023ef3d Compare July 14, 2026 13:30

style: format go files in native module

61dc63c

Signed-off-by: Param <param15.veer.singh@gmail.com>

Xunzhuo requested changes Jul 15, 2026

View reviewed changes

This was referenced Jul 15, 2026

feat: migrate Candle adapter and Rust registry (Phase 2) #2532

Open

Refactor: Modularize native router model bindings and runtime capability contracts #2458

Open

fix: address maintainer feedback on native runtime contract (Phase 1)

ad61a35

Signed-off-by: Param <param15.veer.singh@gmail.com>

Paramveersingh-S force-pushed the feature/native-router-phase1 branch 2 times, most recently from 674ed9a to ad61a35 Compare July 15, 2026 15:36

Merge branch 'main' into feature/native-router-phase1

ffebe06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: introduce native model runtime contract #2530

feat: introduce native model runtime contract #2530
Paramveersingh-S wants to merge 6 commits into
vllm-project:mainfrom
Paramveersingh-S:feature/native-router-phase1

Paramveersingh-S commented Jul 14, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

Xunzhuo left a comment

Uh oh!

Xunzhuo Jul 15, 2026

Uh oh!

Xunzhuo Jul 15, 2026

Uh oh!

Xunzhuo Jul 15, 2026

Uh oh!

Xunzhuo Jul 15, 2026

Uh oh!

Xunzhuo Jul 15, 2026

Uh oh!

Paramveersingh-S commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

Paramveersingh-S commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Key Additions

Uh oh!

netlify Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vllm-semantic-router ready!

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👥 vLLM Semantic Team Notification

📁 src/semantic-router

🎉 Thanks for your contributions!

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Supply Chain Security Report — All Clear

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

Xunzhuo Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Xunzhuo Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Xunzhuo Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Xunzhuo Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Xunzhuo Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Paramveersingh-S commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Paramveersingh-S commented Jul 14, 2026 •

edited

Loading

netlify Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

📁 `src/semantic-router`

github-actions Bot commented Jul 14, 2026 •

edited

Loading