Ultralytics
integrations/sony-imx500/ #17496
giscus[bot]
Bot
Replies: 21 comments 54 replies
|
👋 Hello, thank you for your interest in Ultralytics and exploring integrations with the Sony IMX500! 🚀 For those looking to dive deeper into this topic, the Ultralytics Docs are a great starting point. They contain sections dedicated to various integrations, including Sony IMX500, with detailed instructions on how to export your YOLO models efficiently. If you're here to report a 🐛 bug, please provide a minimum reproducible example. This will help us better understand and assist with the issue you're facing. For general usage, upgrading to the latest version of Ultralytics might resolve some issues you encounter. Make sure your environment is updated with the latest package: pip install -U ultralyticsFor interactive discussions, consider joining the Ultralytics community on Discord 🎧, our Discourse forum, or participate in threads on our Subreddit. Our team will assist you soon, but for now, these resources should help you get started! 😊 EnvironmentsFeel free to run YOLO in any of these verified environments with preinstalled dependencies:
StatusOur continuous integration (CI) ensures all models and integrations, including Sony IMX500, are tested across platforms. Check our CI status here for the latest updates:
|
|
I ran into some python dependency issues when trying to run the 1st example. i started with python 3.12. ultralytics asks for tensorflow 2.12 which seems to clash with the requirements of the converter tools. Downgrading python to 3.11 is not enough. Tensorflow version must be higher than 2.12. I tested on ubuntu 22 machine. Found solution after some testing. Here is the project file. [project]
name = "ultralytics-imx500"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = "~=3.11.0"
dependencies = [
"imx500-converter[pt]==3.14.3",
"model-compression-toolkit==2.1.1",
"sony-custom-layers==0.2.0",
"tensorflow~=2.14.0",
"ultralytics>=8.3.31",
]It could also be good to mention that converter installation at some point might ask for sudo pwd since it requires java. |
|
@alexander-sony thank you for sharing your solution and insights on resolving the dependency issues. It's great to see the community collaborating to overcome challenges. If you have further questions or need additional assistance, feel free to ask here. |
|
Hello Alexander-Sony, root@5c41d2cb95d6:/datasets/dataset# yolo export model=best.pt format=imx PyTorch: starting from 'best.pt' with input shape (1, 3, 416, 416) BCHW and output shape(s) ((1, 3549, 4), (1, 3549, 8)) (5.9 MB) During handling of the above exception, another exception occurred: Traceback (most recent call last): |
|
Hello, I found it myself. Installing openjdk-17-jdk before starting the export prevents the problem. |
|
Thank you for sharing your solution, Jochen! Pre-installing |
|
Hi there it doesnt support for yolov8n-obb model because it shows me |
|
@ARCHANGELOVE it seems you're encountering an issue with the |
|
Yep noted but as i know the box decoding for OBB is differ from AABB because the model convertion using yolov8n.pt works fine and the conversion was successful yolo export model=yolov8n-obb.pt format=imx i face the error when i convert the obb model my ultralytics vertion is in latest aswell |
|
It seems like the |
|
Hello there, Are there any tips on IMX exporting for later YOLO models such as V11? |
|
@marcivanmanalac to export later YOLO models like YOLOv11 to IMX format, you can follow a similar process as mentioned for YOLOv8, using the |
|
Hi Glenn, Yes, that was what I had attempted in the following format but I get a |
|
@marcivanmanalac the error occurs because IMX export currently supports only YOLOv8 detection models and is not implemented for YOLOv11. Please refer to the Sony IMX500 Export guide for supported workflows. YOLOv11-specific export capabilities may require future updates. |
|
There is a problem with the instructions to install picamera2, it seems to create issues for the imx500 examples in https://github.com/raspberrypi/picamera2/tree/main/examples/imx500. Has it been tested with existing imx500 examples in the picamera2 repo? Proper picamera2 installation is with apt, see https://github.com/raspberrypi/picamera2 |
|
@alexander-sony thank you for highlighting this. The installation via |
|
problem with ultralytics version >8.3.34, the instruction: |
|
@alexander-sony the error indicates a mismatch between the input dimensions expected by the model and the data provided. Ensure the input image matches the expected size (default: 640x640) by resizing or specifying |
|
Can we do face recognition(not just detection) with this approach? |
|
@HelloCigar face recognition is out of scope for YOLO models, which are designed for object detection, classification, segmentation, and pose estimation. For facial recognition tasks, consider exploring specialized libraries like FaceNet or Mediapipe. Let me know if you have any YOLO-specific questions! |
|
https://huggingface.co/arnabdhar/YOLOv8-Face-Detection |
|
@HelloCigar using YOLOv8 for face classification (labeling faces with names) can work if your dataset is well-annotated. However, this approach is not true "face recognition" as it lacks the specialized feature extraction and matching capabilities of dedicated face recognition models like FaceNet. The main disadvantage is that YOLOv8 may struggle with nuanced identity recognition compared to models tailored for such tasks. Let me know if you need further clarification! |
|
Great guide and I managed to get my first model trained and converted to run on the imx500 on a raspberry pi4 for some custom object detection. Initially I trained on the yolov8s but when I tried to export it it did not work. I then trained the yolov8n with the same data and it worked fine (doubled checked and the yolov8s defualt does not export either) Is there a way to get the yolov8s models to export so I can run it on the imx500 with the tools or does it only work for the yolov8n? I am new to the Yolov8 models and the imx500 so was just wondering (the s version performed much better than the n version on my dataset) as once I get the yolov8n packaged up on the raspberry PI it seems to be only about 2MB is size so with the input tensors would think it could handle a bigger model as I think the imx500 has 8MB available for the model and input tensors. |
|
@dave-ct thank you for your detailed feedback! Currently, the IMX500 export supports YOLOv8 detection models like YOLOv8n but does not extend to YOLOv8s. This limitation is related to the model's architecture and compatibility with the IMX500 conversion tools. While the IMX500 hardware has sufficient memory, its constraints on model complexity and processing may require further optimizations or updates to support larger models like YOLOv8s in the future. You can stay updated via our documentation or explore adjustments to your dataset to improve YOLOv8n performance. |
|
Hello @dave-ct , I was able to convert the yolov8n model into a suitable format for the IMX500 and run it successfully as well. However, I couldn’t convert any models other than yolov8n. Have you tried converting any models other than yolov8n, and if so, were you successful? |
|
Are there any plans to support yolov8n-pose or any other version of a yolo pose model in the foreseeable future. I have tried using the above tools with yolov8n-pose and I obtained this result: |
|
@awajs currently, the IMX500 export supports only YOLOv8n detection models and not pose models like YOLOv8n-pose. For updates on future support, please stay tuned to the Ultralytics GitHub repository and documentation. |
|
Thanks! |
|
@awajs thank you for your question. The IMX500 export currently only supports YOLOv8n detection models as documented in our Sony IMX500 integration guide. While we don't have an official timeline for pose model support, we welcome community contributions to expand compatibility. For updates, please monitor our GitHub repository and documentation. |
|
hello i ran succesfully the standard yolov8 model converted in imx in my pi ai cam, now i'm wondering where i can modify the parameter of detection, such as confidence, resolution ecc... i don't see anything like the config json of the models tath comes preinstalled with the pi cam dependency |
|
The IMX500 export doesn’t generate a separate configuration file, so detection parameters must be adjusted at inference time in your deployment script or application. |
|
ok, i see, is there some documentation about this topic, and which would be the right way to extrapolate recognition data from the real time detection? |
|
See the Raspberry PI AI Documentation which further down the page explains how to us it for libcamera and picamera2. There are also exmamples of python code on github for python. Hope that helps you. |
|
For adjusting detection parameters like confidence and resolution with your IMX-converted YOLOv8 model on Raspberry Pi, modify these settings directly in the inference script arguments when running See the picamera2 examples for implementation details and our IMX500 export guide for deployment parameters. |
|
in the object detection sample, how do you crop and save the detected obj? assuming only 1 is detected |
|
@HelloCigar to crop and save a detected object from YOLO inference results when only one detection exists, access the bounding box coordinates from from PIL import Image
results = model.predict("image.jpg")
im = Image.open("image.jpg")
box = results[0].boxes.xyxy[0].cpu().numpy() # [x1,y1,x2,y2]
cropped = im.crop(box)
cropped.save("object.jpg")For production use, add conditional checks for detections and handle multiple objects. The detection coordinates remain consistent across export formats like IMX500. |
|
Hello, is there a journal / paper that discusses IMX 500 for yolo ? |
|
@AmndanDwii for specific academic publications on IMX500 with YOLO, we recommend referring to Sony's official technical resources at their AITRIOS developer portal. While Ultralytics doesn't publish academic papers, our implementation details and benchmarks are documented in the IMX500 integration guide and demonstrated in our Edge AI blog post. |
|
Can sony imx run yolo12, does it run the same way as in yolov8? |
|
@AmndanDwii yOLO12 IMX500 support isn't currently available, but the YOLOv8 IMX500 export guide shows our general approach for sensor-optimized deployments. We recommend following the Sony IMX500 Export Guide for YOLOv8 as reference, and stay tuned to our documentation for future YOLO12 updates. |
|
Hello, I got a bunch of warnings during the execution (export_log.txt), but the imx_model seemed to work properly in the end. I then packaged it via IMX500 Packager and test it using the imx500_object_detection_demo.py. |
|
The IMX500 output denormalization details are covered in the Sony IMX500 documentation under the What's Under the Hood? section. For bounding boxes, apply the raw_boxes = results[0].numpy() # raw output tensor for boxes (ordinal=0)
normalized_boxes = raw_boxes * 0.03125 # apply scale factorLet us know if this resolves the coordinate scaling issue! |
|
I think there’s been a misunderstanding. I thought there were other changes to be made besides rescaling. Also, I cannot find the "What's Under the Hood?" section anywhere here https://developer.sony.com/imx500. Could you be more precise about where I should look? |
|
The "What's Under the Hood?" section refers specifically to our Ultralytics IMX500 documentation, not Sony's site. For box decoding, you'll need to handle both scale factors and output tensor ordering. Here's the full post-processing snippet: # After inference
boxes = results[0].numpy() * 0.03125 # scale boxes (ordinal=0)
scores = results[1].numpy() # confidence scores (ordinal=1)
classes = results[2].numpy() # class indices (ordinal=2)
valid = results[3].numpy()[0] # num valid detections (ordinal=3)
# Slice actual detections using valid count
boxes = boxes[:valid]
scores = scores[:valid]
classes = classes[:valid]This accounts for the 4-output-tensor structure documented in our IMX500 integration guide. |
|
Hi @glenn-jocher , thank you very much for all the support. It finally works! There was no need to rescale the boxes by 0.03125. I was missing the --bbox-normalization flag when launching the script. Once I added the flag, everything worked fine. |
|
Great to hear you resolved it! The |
|
Hi, I followed the steps in this page to convert the yolov8n model with imx500 converter and tested it with raspberry ai camera. But I get 29.5 ms for the inference time with 640x640 input tensor size. So I wonder why my inference time is better than your result (58.82ms). Which version of imx500 converter did you use? |
|
Also, how did you measure the inference time? Did you use imx500.get_kpi_info(metadata) or calculate the elapsed time of picam2.capture_metadata()? |
|
@skim-sony congratulations on achieving better inference times than our documented benchmarks! The performance difference could be due to several factors including using a newer IMX500 converter version, firmware optimizations, or hardware differences (Pi 5 vs Pi 4). For measurement methodology, our benchmarks typically use a combination of methods including the IMX500's native profiling via It's great to see improved performance in newer implementations - would you mind sharing which converter version and Raspberry Pi model you're using? This could help other community members optimize their deployments. |
|
Hi, I have used Pi5 and tried both 3.14.3 and 3.16.1 for imx500-converter. And both produces 29.5ms for inference time. It seems like you also use Pi5, so I wonder why our results are different. I'd like to reproduce your results 58.82 ms to learn what improves the inference time. So it would be helpful if you could share some info how to sync with your environment. Thanks! |
|
@skim-sony that's interesting you're seeing better performance! The discrepancy could be due to several factors:
For replicating our methodology, we used:
Ultimately, faster performance is a positive outcome! Your 29.5ms results demonstrate how the platform continues to improve. If you'd like to share more details about your implementation, the community would certainly benefit. |
|
Hello, per picamera2 docs, creating apps with "a camera window embedded into the GUI is through Qt", is there no other way to embed the video feed in other GUI frameworks? |
|
In the official IMX500 documentation provided by Sony (https://developer.sony.com/tinyml-23/developer-tools/documentation/imx500-evaluation-kit-user-guide?version=2023-11-28&progLang=), they describe the process of generating the network.fpk and network_info.txt files that will be used to deploy the model to IMX500, using the Packager.sh script. However, when I attempt to apply the same procedure to the packerOut.zip file I created by following the tutorial on https://docs.ultralytics.com/integrations/sony-imx500, I am unable to run the model on the IMX500. Also I couldn't run your implementation because it says that some packages are not found which I believe is related to the version of the OS. Currently I use the Pi OS that is provided by SONY for compatibility issues. I could run some other deep learning models following the steps in the (https://developer.sony.com/tinyml-23/developer-tools/documentation/imx500-evaluation-kit-user-guide?version=2023-11-28&progLang=), but I believe, there must be something different related to the creation of packerOut.zip file. Could you assist in resolving this issue with the conversion process? @glenn-jocher |
|
One thing that would be very helpful is to show how to do this on something other than the target platform. i'm trying to get a small detection model to run on a rpi zero2 and hate to have to download all of my training data to the zero to do the quantization. Idea: Have a step that allows collection of the "target_platform_capabilities" information using Sony's model compression toolkit you are using under the hood, pickle it, and then use that object to drive the model compression on a larger machine. I could probably dig in and try to follow your code and follow the "imx" part of your export process... |
|
Much thanks for posting this guide. Thank you. I am having very solid success running the standard detection models right out of the box on the RpiZero 2 which is my target platform. I'm actually amazed at the power draw (ran detection/streaming for about 175 minutes on a 4400mahr battery using about 60% of its capacity). Also, very solid top/htop scores showing less than 1/8 of CPU utilization. Memory is still such a problem on a rpiz2 but not any more so than what is being used with streaming only on the pi. Still have about 50% of memory left over as I recall. I've got screen shots of top screens if needed. But, I need to get a trained model working on my RpiZ2 and am running into problems. As the guide seemed to indicate (or perhaps Sony's), I need to do most of the quantization on my target platform. I assumed that meant my zero. I have a very good model trained with yolov11m or large. I've tried to reduce that to a nano but I'm getting too much loss of MAP scores and too much false positive rates. So, am now trying to get a yolov11s to work. On my quantization process (it was not clear how to do this with a custom model) but am first loading the yolov11s.pt and then loading my best.pt on top of that. Ultralytics successfully loads my data, gives me a warning prompt about my 290 images (< 300 not ideal) and then another about processing speed with 1 batch. The program then goes off the rails with no UI messages. I assume I'm running out of memory on the piz2. So, my next steps seem to be either:
I guess another option would be to spend more time understanding Sony's under the hood tools to optimize those settings? Would appreciate some guidance here down the right path. |
|
I've made considerable progress by moving conducting this model compression process with a Container using dtcooper/raspberrypi-os:python3.11 after considerable problems with dependencies. Now getting a container most of the way through the process of compressing my yolov11s model and its trained best.pt weights. At the end of the process - I get a simple "killed" after the Calculation of quantization parameters step. I see memory spiking close to 100% on top and think I'm hitting OOM with my pi4 8gb. Would appreciate any suggestion around this but think it probably pushes me over the edge to order my pi5? Or do I try to train on the yolov8 or yolov11n? I've not had very good success with the nano models. I've convinced myself the tradeoffs to the small is well worth it but that may be incompatible with this process? |
|
I opened an issue: #22512 Just tried the export again with the fraction=0.25 argument and got to "Optimal Solution Found" with objective measure of 0.94, then died with error: Error x IMX: export failure 12662.7s: No module named 'onnxscript' |
|
pytorch/pytorch#166352 -- adding to my requirements.txt pointed to by Containerfile. |
|
@leeprevost great progress. Two quick notes and a fix for the new error: IMX export/quantization does not need to run on the target; the exporter cross-compiles for IMX500 by loading Sony’s Target Platform Capabilities for The “No module named 'onnxscript'” failure is a missing dependency; install it and align the IMX toolchain pins used by the exporter (see the export_imx reference): pip install -U onnxscript "onnx>=1.17,<1.18" "imx500-converter[pt]>=3.16.1" \
"model-compression-toolkit>=2.4.1" "sony-custom-layers>=0.3.0" "edge-mdt-tpc>=1.1.0" \
"pydantic<=2.11.7"Let’s continue the deep-dive in the issue you opened for traceability. |
|
Ok -- @glenn-jocher: am moving this to my opened issue. Still trying this on my pi4 but can move to my dev machine with plenty of RAM + Cuda. Got further but still died with a torch error -- will post there. |
|
Thanks—let’s continue in the issue for traceability. On the larger Linux host, try GPU-assisted export to reduce RAM pressure and speed up calibration; if needed, temporarily lower yolo export model=path/to/best.pt format=imx data=path/to/data.yaml device=0 imgsz=640
# if memory is still tight during calibration
yolo export model=path/to/best.pt format=imx data=path/to/data.yaml device=0 imgsz=512For a concise list of export options and behavior, see the Export mode docs at Ultralytics YOLO — Export. |
|
where is the best place to get support on the IMX500. I am running into multiple issues: i. my first yolo11n model built fine and exported to imx500, no problem, but my second model with a LOT more training images build fine but crashed the imx500 export. ii. the output of the imx500 during inference is giving a lot of none outputs between solid recognitions. it seems like there is something interfering with or disrupting the inference outputs (e.g., good output, none, none, none, none, good output, ...). where is the best place to get help with these issues? |
|
Please continue the discussion here so the Ultralytics team can help; for the export crash, share the exact export command, full text traceback, and output from |
Uh oh!
There was an error while loading. Please reload this page.
integrations/sony-imx500/
Learn to export Ultralytics YOLOv8 models to Sony's IMX500 format to optimize your models for efficient deployment.
https://docs.ultralytics.com/integrations/sony-imx500/
All reactions