[ACL 2026]Scaling Behaviors of LLM Reinforcement Learning Post-Training

[ACL 2026 Main Conference] An Empirical Study in Mathematical Reasoning

This repository contains the implementation and experimental code for our paper: "Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning".

📖 Overview

This research systematically investigates scaling behaviors in RL-based post-training for large language models, with a focus on mathematical reasoning. Through 63 experiments across the full Qwen2.5 dense model series (0.5B to 72B), we characterize how model scale, data volume, and computational budget interact to shape performance.

Key Contributions

• 🔬 Comprehensive Scaling Analysis: First systematic study of RL post-training scaling laws across 0.5B-72B parameters for mathematical reasoning
• 📊 Predictive Power Laws: Established robust relationships between model performance, compute, and data
• 🎯 Learning Efficiency Insights: Discovered efficiency saturation trends in larger models
• 💾 Data Reuse Strategy: Demonstrated effectiveness of data reuse in constrained settings (up to 25× reuse with minimal degradation)

---

🎯 Main Findings

🔑 Finding 1: Predictive Scaling Laws

The relationship between test loss L (where L = 1 - Pass@1) and training resource X (Either compute C or Data D) follows robust power-law patterns:

$$ \log L(N, X) = -k(N) \cdot \log X + E(N) $$ where $$ k(N) = \frac{K_{\mathrm{max}}}{1 + \frac{N_0}{N}} $$

It enables accurate PREDICTIONS for both unseen model sizes and remaining training trajectories.

Inter-model Extrapolation in compute scenario: Fitted on 0.5B-32B, extrapolated to 72B

Intra-model Prediction in compute scenario: Predict remaining trajectory from early steps

🔑 Finding 2: Learning Efficiency Saturation

Larger models consistently achieve better learning efficiency during RL post-training. As shown in the figures below, scaling up model size leads to substantial improvements in compute efficiency. But our analysis reveals that the learning efficiency coefficient k(N) exhibits a saturation trend, where the marginal gains in learning efficiency gradually diminish.

Learning Efficiency Saturation: k(N) follows a saturation curve, with diminishing marginal returns as N increases

This saturation pattern becomes particularly evident beyond 32B parameters, which may introduce a trade-off between model scale and training steps under limited compute budgets.

🔑 Finding 3: Data Reuse Effectiveness

In data-constrained settings, repeated reuse of high-quality data is effective. Final performance is primarily governed by total optimization steps rather than sample uniqueness.

Data Reuse Analysis: Performance remains stable up to τ=25 reuses (Base & Instruct)

• Performance stable up to τ = 25 (25× reuse)
• Moderate overfitting only appears at extreme reuse factors (τ = 100)

---

🚀 Quick Start

1. Environment Setup

# System Requirements
# - Python 3.12
# - CUDA 12.4
# - PyTorch 2.6.0

# Clone repository
git clone https://github.com/tanzelin430/Mathematical-Reasoning-RL-Scaling-Law.git
cd Agentic-RL-Scaling-Law

# Install dependencies
pip install -r requirements.txt
pip install -e .[gpu,test,math,vllm]

2. Data Preparation

# Download and prepare the guru-RL-92k mathematics dataset
python src/data/prepare_math_by_difficulty_full.py

# The script will:
# - Parse Qwen2.5-7B pass rates
# - Classify problems by difficulty (easy: rate ≥ 0.3, hard: rate < 0.3)
# - Generate curriculum-ordered dataset (53,904 train + 500 test)

Dataset Structure:

• Training: 53,904 math problems (ordered by increasing difficulty)
• Validation: 500 held-out math problems (stratified sampling)
• Evaluation: Multi-domain benchmarks (AIME, GSM8K, AMC, HumanEval, etc.)

3. Run Training

# Train different model sizes (0.5B to 72B)
bash scripts/train/run_grpo_qwen2.5_0.5b_math_full_cl.sh   # 0.5B
bash scripts/train/run_grpo_qwen2.5_1.5b_math_full_cl.sh   # 1.5B
bash scripts/train/run_grpo_qwen2.5_3B_math_full_cl.sh     # 3B
bash scripts/train/run_grpo_qwen2.5_7b_math_full_cl.sh     # 7B
bash scripts/train/run_grpo_qwen2.5_14B_math_full_cl.sh    # 14B
bash scripts/train/run_grpo_qwen2.5_32B_math_full_cl.sh    # 32B
bash scripts/train/run_grpo_qwen2.5_72B_math_full_cl.sh    # 72B

Training Configuration:

• Algorithm: GRPO (Group Relative Policy Optimization)
• Batch Size: 512
• Rollout: 8 responses per prompt
• Learning Rate: 1e-6
• KL Coefficient: 0.001
• Max Sequence Length: 2048 (input) + 4096 (output)
• Framework: VeRL with FSDP distributed training

📊 Experimental Setup

Models

• Architecture: Qwen2.5 series (shared architecture across all sizes)
• Sizes: 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B parameters
• Variants: Both Base and Instruct models

Training Data

• Source: guru-RL-92k mathematics subset (54.4k problems)
• Curriculum Learning: Problems sorted by Qwen2.5-7B pass rate (easy → hard)
• Train/Test Split: 53,904 / 500 (stratified by difficulty)

Evaluation Benchmarks

Domain	Benchmarks
Math	AIME2024, AMC2023, GSM8K, MATH-500, Held-out data
Code	HumanEval
Logic	Zebra Puzzle
STEM	SuperGPQA

---

📂 Repository Structure

Agentic-RL-Scaling-Law/
├── verl/                          # VeRL RL training framework
│   ├── trainer/                   # PPO/GRPO training logic
│   ├── workers/                   # FSDP distributed workers
│   └── utils/                     # Reward scoring, datasets
├── scripts/
│   ├── train/                     # Training scripts for different sizes
│   │   ├── run_grpo_qwen2.5_0.5b_math_full_cl.sh
│   │   ├── run_grpo_qwen2.5_7b_math_full_cl.sh
│   │   └── ...
│   └── train_data_check/          # Data validation scripts
├── src/
│   ├── data/                      # Data preprocessing utilities
│   │   ├── pre_verl.py            # Convert to VeRL format
│   │   └── prepare_math_by_difficulty_full.py
│   └── plot/                      # Visualization scripts
├── analysis/                      # Scaling law fitting and analysis code
├── data/                          # Datasets
│   ├── math_curriculum/           # Difficulty-ordered math
│   └── guru_verl/online_eval/     # Multi-domain validation
├── outputs/                       # Generated figures and results
└── SandboxFusion/                 # Code execution sandbox

💡 For detailed information on how we analyze the scaling laws and generate the figures in this paper, please refer to the analysis/ folder.

---

📝 Citation

If you find this work helpful, please consider citing:

@inproceedings{tan-etal-2026-scaling,
    title = "Scaling Behaviors of {LLM} Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning",
    author = "Tan, Zelin  and
      Geng, Hejia  and
      Yu, Xiaohang  and
      Zhang, Mulei  and
      Wan, Guancheng  and
      Zhou, Yifan  and
      He, Qiang  and
      Xue, Xiangyuan  and
      Zhou, Heng  and
      Fan, Yutao  and
      Li, Zhong-Zhi  and
      Zhang, Zaibin  and
      Zhang, Guibin  and
      Zhang, Chen  and
      Yin, Zhenfei  and
      Torr, Philip  and
      Bai, Lei",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1444/",
    pages = "31300--31319",
    ISBN = "979-8-89176-390-6",
    abstract = "While scaling laws for large language models (LLMs) during pre-training have been extensively studied, their behavior under reinforcement learning (RL) post-training remains largely unexplored. This paper investigates the scaling behavior of Large Language Model (LLM) reinforcement learning post-training, focusing on mathematical reasoning. Through experiments across the Qwen2.5 series (0.5B to 72B), we characterize how model scale, data, and compute interact. Our analysis yields four key findings: 1. Larger models consistently demonstrate superior compute and data efficiency. 2. The relationship between model performance and training resources follows a **predictive power-law** across both base and instruction-tuned models. 3. RL learning efficiency exhibits a latent **saturation trend** with increasing model scale. 4. In data-constrained regimes, performance is primarily driven by the **total volume of training data** rather than sample uniqueness. These results offer practical guidelines for scaling reasoning capabilities through reinforcement learning post-training."
}

---

🙏 Acknowledgments

• VeRL Framework from Verl
• Guru-RL-92k Dataset from LLM360
• Qwen2.5 Models from Qwen

---

📧 Contact

For questions or collaborations, please contact:

• Chen Zhang: zhangchenzc@mail.ustc.edu.cn
• Zhenfei Yin: jeremyyin@robots.ox.ac.uk
• Zelin Tan: tanzl@mail.ustc.edu.cn