Problem
When a model is fine-tuned with TRL's SFTTrainer, the saved generation_config.json can end up with pad_token_id set to a token that is also in the eos_token_id list. This causes model.generate() to emit EOS immediately, producing empty completions.
Reproduction
We fine-tuned nvidia/NVIDIA-Nemotron-Nano-9B-v2 with TRL SFTTrainer. The base model has:
"pad_token_id": 0,
"eos_token_id": 12
After SFT training, the saved generation_config.json had:
{
"eos_token_id": [2, 11, 12],
"pad_token_id": 12
}Token 12 (<|im_end|>) is both pad AND eos. When model.generate() is called with pad_token_id=12, the model sees token 12 as EOS and stops immediately. HumanEval scored 0.0% (0/164) — every completion was empty.
The base model scored 76.2% (125/164) with the same eval script. The SFT model itself learned correctly (loss 0.94, token accuracy 78%) — the 0% is purely a config bug.
Root Cause
TRL's SFTTrainer sets tokenizer.pad_token = tokenizer.eos_token when no pad token exists. For Nemotron, eos_token is <|im_end|> (token 12), which is also in the eos_token_id list [2, 11, 12]. The trainer saves this collision into generation_config.json.
GenerationConfig.validate() checks various parameter combinations (do_sample vs top_p, etc.) but does not check for pad/eos collision.
Proposed Fix
Add a validation check in GenerationConfig.validate():
if self.pad_token_id is not None and self.eos_token_id is not None:
eos_ids = self.eos_token_id if isinstance(self.eos_token_id, list) else [self.eos_token_id]
if self.pad_token_id in eos_ids:
warnings.warn(
f"pad_token_id ({self.pad_token_id}) is in eos_token_id list ({eos_ids}). "
f"This may cause model.generate() to stop immediately. "
f"Consider setting pad_token_id to a token not in eos_token_id."
)This would warn users at config load time, before they waste hours running evals that produce 0%.
Environment
- transformers: 5.16.0.dev0 (main branch)
- trl: 1.10.0
- Model: nvidia/NVIDIA-Nemotron-Nano-9B-v2 (NemotronH hybrid Mamba-2 + Attention)
- GPU: NVIDIA GB10 (119GB VRAM)
Related
Problem
When a model is fine-tuned with TRL's SFTTrainer, the saved
generation_config.jsoncan end up withpad_token_idset to a token that is also in theeos_token_idlist. This causesmodel.generate()to emit EOS immediately, producing empty completions.Reproduction
We fine-tuned
nvidia/NVIDIA-Nemotron-Nano-9B-v2with TRL SFTTrainer. The base model has:After SFT training, the saved
generation_config.jsonhad:{ "eos_token_id": [2, 11, 12], "pad_token_id": 12 }Token 12 (
<|im_end|>) is both pad AND eos. Whenmodel.generate()is called withpad_token_id=12, the model sees token 12 as EOS and stops immediately. HumanEval scored 0.0% (0/164) — every completion was empty.The base model scored 76.2% (125/164) with the same eval script. The SFT model itself learned correctly (loss 0.94, token accuracy 78%) — the 0% is purely a config bug.
Root Cause
TRL's SFTTrainer sets
tokenizer.pad_token = tokenizer.eos_tokenwhen no pad token exists. For Nemotron,eos_tokenis<|im_end|>(token 12), which is also in theeos_token_idlist[2, 11, 12]. The trainer saves this collision intogeneration_config.json.GenerationConfig.validate()checks various parameter combinations (do_sample vs top_p, etc.) but does not check for pad/eos collision.Proposed Fix
Add a validation check in
GenerationConfig.validate():This would warn users at config load time, before they waste hours running evals that produce 0%.
Environment
Related