Add DAPO advantage estimator and Soft Overlong support#362
Conversation
Signed-off-by: kaiyuan <kyxiezju@163.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) algorithm, adding its documentation, configuration arguments, default recipe validation, and the implementation of Soft Overlong Punishment in reward post-processing. The review feedback points out potential TypeError crashes during argument validation and rollout if rollout_max_response_len is not specified while DAPO or Soft Overlong is enabled, and suggests adding explicit checks and clearer error messages to prevent these issues.
| if args.soft_overlong_cache is None: | ||
| args.soft_overlong_cache = max(1, int(args.rollout_max_response_len) // 4) |
There was a problem hiding this comment.
If --rollout-max-response-len is not specified (i.e., it is None), calling int(args.rollout_max_response_len) will raise a TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'. Since DAPO defaults to enabling Soft Overlong, we should check if args.rollout_max_response_len is None and raise a clear ValueError to guide the user.
if args.soft_overlong_cache is None:
if args.rollout_max_response_len is None:
raise ValueError("DAPO requires --rollout-max-response-len to be set to enable Soft Overlong.")
args.soft_overlong_cache = max(1, int(args.rollout_max_response_len) // 4)| if args.soft_overlong_cache is not None and args.soft_overlong_cache < 0: | ||
| raise ValueError(f"--soft-overlong-cache must be >= 0, got {args.soft_overlong_cache}.") |
There was a problem hiding this comment.
If the user explicitly sets --soft-overlong-cache to a positive value but does not specify --rollout-max-response-len, a TypeError will occur later during rollout when calculating L_max. We should validate this condition here and raise a clear ValueError.
if args.soft_overlong_cache is not None:
if args.soft_overlong_cache < 0:
raise ValueError(f"--soft-overlong-cache must be >= 0, got {args.soft_overlong_cache}.")
if args.soft_overlong_cache > 0 and args.rollout_max_response_len is None:
raise ValueError("--rollout-max-response-len must be set when --soft-overlong-cache is enabled.")
|
NPU tests: |
Signed-off-by: kaiyuan <kyxiezju@163.com>
1 participant
Summary
--advantage-estimator dapowith default Clip-Higher, token-level loss, dynamic sampling, and Soft Overlong--soft-overlong-cacheand apply length penalty in rollout reward post-processing before group normTest plan
pre-commiton changed filespytest tests/test_megatron_argument_validation.py--advantage-estimator dapoand--over-sampling-batch-size > --rollout-batch-size