Combining multiple yolo models together with shared backbone and different heads

[yucath]

Hi, I am trying to combine multiple yolo models (same type - for eg yolov8s-seg, yolov8s-obb, yolov8s). As the models share the common backbone (is that right?), I am trying to separate the backbone and heads of differtn models and tryign to use only one backbone and the different heads (from the full model which was trained separately - and only use head from those). The idea is to reuse the backbone and train wiht more generalised images that can be used across multiple tasks and head is only responsible for some specific classes from the outputs of those backbones. I am exporting backbone adn head separately as .plan files and using triton server to serve the model with a router which tells how to combine those together.

I am able to do that and get results, but the results do not make sense to me. I think there is something going on. My question is: is this strategy fundamentally wrong? and is there anything that needs to be done to allow for this?

Open to any suggestions or pointers that can help.

[UltralyticsAssistant]

👋 Hello @yucath, thank you for starting this thoughtful discussion 🚀 This is an automated response — an Ultralytics engineer will follow up shortly to assist further.

We recommend a visit to the Docs for architecture and usage references where many Python and CLI examples may help frame your approach.

If this is a 🐛 Bug Report related to unexpected behavior when mixing backbones/heads or during export/serving, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question or design discussion, please share:
-Exact model variants and versions used (e.g., yolov8s-seg, yolov8s-obb, yolov8s)
-Training logs, hyperparameters, and any modifications to model configs
-Export details (.plan generation steps, toolchain versions) and Triton model repository structure/router configuration
-Example inputs/outputs that illustrate the mismatch you’re seeing
-And verify you’re following our Tips for Best Training Results

Join the Ultralytics community where it suits you best. For real-time chat, head to Discord 🎧. Prefer in-depth discussions? Check out Discourse. Or dive into threads on our Subreddit to share knowledge with the community.

Upgrade

Before proceeding, please upgrade to the latest ultralytics release and verify all requirements in a Python>=3.8 environment with PyTorch>=1.8. This helps ensure any known issues are already resolved:

pip install -U ultralytics

Environments

YOLO can be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

To help us provide targeted guidance on your shared-backbone/different-heads setup with Triton, please include the details above along with any minimal configs or code snippets demonstrating how you’re wiring the backbone output to each head. This will help us quickly validate assumptions and suggest next steps 🔍

[glenn-jocher]

Great question. Heads trained separately are strongly co-adapted to their original backbone+neck, so wiring those heads to a different shared trunk usually yields inconsistent outputs. The approach isn’t inherently wrong, but it only works if each head receives exactly the features it was trained on (same P3/P4/P5 order, channels, and strides) and if task-specific branches are honored—segmentation needs prototypes and per-scale mask coefficients, and OBB adds angle logits with rotated-box decoding. The most reliable path is to define a single model with one shared trunk and multiple heads in a custom YAML, train or at least fine-tune the heads jointly on that trunk, then export once to ONNX→TensorRT and serve a single engine with multiple outputs; see Model YAML guidance in Model YAML Configuration, head specifics in the OBB head and YOLO Segmentation head, how Detect/Segment/OBB consume the ch feature tuple in the parse_model reference, and deployment notes in the Triton guide.