MO-GRPO

This repository contains implementations of MO-GRPO for multi-objective optimization across different domains including machine translation, control task, and bandit problems.

Advantage Normalization (GRPO vs. MO-GRPO)

GRPO constructs a group-normalized advantage from a set of sampled sentences $\mathbf{o}={o_1,\dots,o_G}$ for the same prompt $q$. When using $K$ reward models, the standard GRPO implementation first aggregates the rewards into a single scalar and then applies normalization within the group:

MO-GRPO

MO-GRPO instead normalizes each reward dimension independently (per-objective z-score), and then sums the normalized scores:

Intuition.

• Standard GRPO normalizes after summing rewards, so objectives with larger scale/variance can dominate.
• MO-GRPO equalizes scales across objectives before aggregation, making optimization less sensitive to reward magnitude mismatch and improving robustness in multi-objective settings.

The following figure summarizes the key difference between the sum-then-normalize strategy in standard GRPO and the normalize-then-sum strategy in MO-GRPO.

Figure: GRPO aggregates rewards first and normalizes once, whereas MO-GRPO normalizes each reward dimension independently before aggregation.

Installation

Prerequisites

• Python 3.12
• cuda:12.6.1-devel-ubuntu22.04

Setup

Install dependencies for each experiment type:

python3 -m venv env && source env/bin/activate
# system deps + python deps
bash setup.sh

For Bandit Experiments:

Note: Bandit experiments do not require CUDA/GPU. You can run them on CPU-only environments.

cd bandit
python plot_grpo_rewards.py

For Machine Translation (↓Example: Running GRPO):

cd language_experiment
bash get_wmt.sh
bash run_mt_grpo.sh

If you want to compare MO-GRPO to Dr.GRPO:

pip install -r requirements_dr.txt
bash run_mt_dr_grpo.sh

For Control TASK:

cd mo-gymnaisum
python grpo_reacher.py \
    --env_id "mo-reacher-v4" \
    --variant "orig" \
    --weights 1.0,1.0 \
    --steps 10000 \
    --group_size 8