SAM3: Passing visual prompt from one image to another

[diophantus7]

Hello,
I have a series of images (with some overlap). I would like to annotate a single image with a (couple) visual prompt and find similar objects in the consecutive images. I tried the SAM3VideoPredictor but I do not want to have the tracking of the object. Can anyone point me to a solution or map out a possible way to solve this?

[UltralyticsAssistant]

👋 Hello @diophantus7, thank you for your interest in Ultralytics 🚀! This is an automated response to help get your discussion moving, and an Ultralytics engineer will also assist you soon.
We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.
Since this looks like a custom workflow ❓ question around prompting across overlapping images, please provide as much information as possible, including:

• Your current model or predictor setup
• A short description of the image sequence and overlap
• Example images or screenshots
• The prompts you are using
• What output you expect vs. what you are seeing
• Any logs or minimal code snippets you have tried
  If this turns out to be a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.
  If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.
  Join the Ultralytics community where it suits you best. For real-time chat, head to Discord 🎧. Prefer in-depth discussions? Check out Discourse. Or dive into threads on our Subreddit to share knowledge with the community.

Upgrade

Upgrade to the latest ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8 to verify your issue is not already resolved in the latest version:

pip install -U ultralytics

Environments

YOLO may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

For non-tracked image-to-image prompting, SAM3VideoPredictor isn’t the right fit; the documented cross-image workflow today is YOLOE visual prompting with refer_image, while SAM 3 currently documents exemplar prompts for concept segmentation on the image being processed:

import numpy as np
from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

model = YOLOE("yoloe-26l-seg.pt")
visual_prompts = dict(
    bboxes=np.array([[x1, y1, x2, y2]]),
    cls=np.array([0]),
)

results = model.predict(
    ["img2.jpg", "img3.jpg", "img4.jpg"],
    refer_image="img1.jpg",
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
)

If you want to stay strictly in SAM 3, you’d currently need to run each image independently rather than use the video tracker.

[Erfan-ram]

Hi @diophantus7,

I recently ran into the exact same challenge. I needed to find a specific object across hundreds of independent images with SAM 3's pixel-level accuracy, and while YOLOE is great, it didn't quite give me the fine mask details that SAM 3 provides.

While @glenn-jocher is completely right that the standard SAM 3 API natively only supports exemplar prompting on the same image, there is a very clever workaround to bypass this limitation without using the video tracker. It's called the "Shared Canvas" (Spatial Concatenation) strategy.

The Core Idea:

Instead of trying to pass features between different images, you can trick SAM 3 by putting both images into the same frame:

1. You dynamically stitch your reference image and your target image side-by-side into a single array (a shared canvas).
2. You pass this concatenated image to the standard SAM3SemanticPredictor.
3. You provide the bounding box coordinates from the reference side of the canvas.
4. Since SAM 3 is designed to find similar visual concepts within the same image, it will naturally find and segment your object on the target half of the canvas.
5. Finally, you just split the resulting mask and resize it back to your target image's original dimensions.

The Ready-to-Use Solution:

You don't even need to write this pipeline from scratch. One of the authors/collaborators in the Ultralytics ecosystem, WongKinYiu (author of YOLOv7), actually built a dedicated wrapper around SAM 3 using this exact logic!

It’s called FSS-SAM3 (Few-Shot Semantic Segmentation Meets SAM3). You can find the repository here:
👉 [WongKinYiu/FSS-SAM3]

It handles the image concatenation, scaling, and negative/positive prompt mining out of the box, allowing you to use a single visual prompt across as many consecutive or independent images as you want, strictly using the SAM 3 engine.

Hope this helps speed up your workflow!

some wrapper on old implementation :
https://github.com/aim-uofa/matcher
https://github.com/zrrskywalker/personalize-sam

i will write my own structures to do the same but in simpler term . ask me if you need a test code here.