Need Guidance for Training Custom YOLO Model with Better Accuracy

[tedo001]

Hello Ultralytics Team and Community,

I’m currently working on a custom object detection project using YOLO and wanted some guidance regarding improving model accuracy.

Current Setup:
Dataset: Custom labeled images
Model: YOLOv8 / YOLO11
Training Epochs: 100
Image Size: 640
Hardware: GPU / Local PC

Questions:
What are the best augmentation settings for improving real-world detection?
How can I improve small object detection performance?
Is transfer learning recommended for niche datasets?
Any tips for reducing false positives?
Goal:
I want to deploy this model into a real-world application with high speed and good precision.

[UltralyticsAssistant]

👋 Hello @tedo001, thank you for your interest in Ultralytics 🚀! This is an automated response to help get things moving quickly, and an Ultralytics engineer will also assist you soon.
We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.
Since this is a custom training ❓ question about improving accuracy for a YOLO11 model, please provide as much information as possible, including dataset image examples, class distribution, validation results, confusion matrix, sample predictions, and training logs, and verify you are following our Tips for Best Training Results.
If you run into a specific 🐛 issue while training or validation, please provide a minimum reproducible example to help us debug it.
Join the Ultralytics community where it suits you best. For real-time chat, head to Discord 🎧. Prefer in-depth discussions? Check out Discourse. Or dive into threads on our Subreddit to share knowledge with the community.

Upgrade

Upgrade to the latest ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8 to verify your issue is not already resolved in the latest version:

pip install -U ultralytics

Environments

YOLO may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[tedo001]

sir ! I used your yolo26x and now accuracy is so good at detection!!

[glenn-jocher]

Good to hear yolo26x improved detection — before deployment, validate on a fully unseen test split and tune conf on real-world images to balance missed detections vs false positives, and if you want to optimize speed next we can help using the Benchmark mode guide.

[tedo001]

I think currently I am so addicted to ultralytics 😅.

[glenn-jocher]

Glad it’s working well now — credit to the YOLO community and Ultralytics team; if you want, the next practical step is to benchmark yolo26x against smaller variants on your target hardware so you can lock in the best speed/accuracy tradeoff before deployment.

[tedo001]

okay sir!!

[vincere-mori]

A few concrete things that actually move the needle:

Small objects: bump imgsz to 1280 if you have the VRAM, that's usually the single biggest win. Also enable copy_paste augmentation and run the auto-anchor check to verify your anchor sizes cover your object scale.

Augmentations for real-world deployment: keep mosaic on, but lower degrees and shear if your objects have a natural orientation in the deployment images. Aggressive geometric augmentations hurt when the test distribution is mostly upright/normal angles.

Transfer learning: yes, always start from COCO pretrained. If you have less than roughly 500 images per class, freeze the backbone for the first 20 epochs using freeze=10 in your training args. It prevents the pretrained features from getting destroyed early on.

False positives: first check what they actually look like visually. If they're structurally similar to your positives, you need hard negatives in the training set. Run inference on your val set, pull the false positives, label them correctly (or as background), and fold them back in. Adjusting the confidence threshold at inference time helps short-term but does not fix the root cause.

[glenn-jocher]

Good suggestions above overall, with one important update: current Ultralytics YOLO models are anchor-free, so I’d skip anchor tuning and instead start from a pretrained yolo26s.pt or yolo26m.pt, follow the two-stage workflow in the fine-tuning guide (freeze=10 for a short first stage, then unfreeze with lower lr0), and for small objects prioritize higher imgsz or tiling plus hard-negative images to reduce false positives as outlined in the small-object accuracy guide; if you share your class count and sample image size, we can suggest a tighter starting config.