DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucination

TMLR 2025

This repository contains the reference code for the paper DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations .

🎯 Project web page |

Paper |

🤗 HuggingFace Model |

Install Packages

conda create -n dadpo python=3.10 -y

conda activate dadpo

pip install -e .

Training data

Download ShareGPT4V from here

Download COCO from here

Download dataset annotation from here

Extract data from ShareGPT4V and organize the images as follows:

Image_root
├── coco/
    ├──train2017/
├── llava/
    ├──llava_pretrain/
├── sam/
├── share_textvqa/
    ├──images/
├── web-celebrity/
    ├──images/
├── web-landmark/
    ├──images/
├── wikiart/
    ├──images/

Train DADPO

bash scripts/da_dpo/7B_DADPO.sh

Acknowledgement

The project is built on top of the amazing multimodal large language model LLaVA, RLHF package trl, DPO for multimodal learning Silkie, visual contrastive decoding VCD and BPO. Thanks for these great work!

If you find our work useful for your research or applications, please cite using this BibTeX:

@article{qiu2026dpo,
  title={DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations},
  author={Qiu, Longtian and Ning, Shan and Zhang, Chuyu and Sun, Jiaxuan and He, Xuming},
  journal={arXiv preprint arXiv:2601.00623},
  year={2026}
}