CI single GPU regressions: 08/13 vs 08/14 diff (42 findings)

[ydshieh]

@ydshieh here is the diff of single GPU failures between 08/13 and 08/14 CI runs, covering all error types for models_* keys.

Status legend:

• NEW — test not present in 08/13 single GPU failures (may exist in 08/13 multi GPU)
• CHANGED — test present in both days but trace content changed

Status	Model	Test	Error type	Prev Error type
CHANGED	bamba	test_simple_batched_generate_with_padding	OutOfMemoryError	OutOfMemoryError
CHANGED	bamba	test_simple_generate	OutOfMemoryError	OutOfMemoryError
CHANGED	cohere2_vision	test_model_integration_generate_chat_template	OutOfMemoryError	OutOfMemoryError
CHANGED	deepseek_v2	test_deepseek_v2_lite	OutOfMemoryError	OutOfMemoryError
CHANGED	deepseek_v2	test_logits_eager	OutOfMemoryError	OutOfMemoryError
CHANGED	emu3	test_model_generation	OutOfMemoryError	OutOfMemoryError
CHANGED	emu3	test_model_generation_batched	OutOfMemoryError	OutOfMemoryError
CHANGED	emu3	test_model_generation_multi_image	OutOfMemoryError	OutOfMemoryError
CHANGED	flava	test_inference	AssertionError: string !=	AssertionError: string !=
NEW	gemma	test_model_7b_fp16_static_cache	AssertionError: Lists differ	N/A
CHANGED	gemma4	test_export_text_only	OutOfMemoryError	OutOfMemoryError
CHANGED	glm	test_model_9b_fp16	OutOfMemoryError	OutOfMemoryError
CHANGED	glm	test_model_9b_sdpa	OutOfMemoryError	OutOfMemoryError
CHANGED	glm4_moe	test_compile_static_cache	OutOfMemoryError	OutOfMemoryError
CHANGED	glm4_moe_lite	test_compile_static_cache	OutOfMemoryError	OutOfMemoryError
CHANGED	llama4	test_model_17b_16e_batch	OutOfMemoryError	OutOfMemoryError
CHANGED	llama4	test_model_17b_16e_fp32	OutOfMemoryError	OutOfMemoryError
CHANGED	llava	test_pixtral	OutOfMemoryError	OutOfMemoryError
CHANGED	llava	test_pixtral_4bit	OutOfMemoryError	OutOfMemoryError
CHANGED	mamba2	test_batched_equivalence_with_cache	OutOfMemoryError	OutOfMemoryError
CHANGED	mamba2	test_batched_equivalence_without_cache	OutOfMemoryError	OutOfMemoryError
CHANGED	mamba2	test_simple_generate	OutOfMemoryError	OutOfMemoryError
CHANGED	mimi	test_integration	AssertionError: np.False_	AssertionError
CHANGED	mimi	test_integration_longform	AssertionError: np.False_	AssertionError
NEW	minicpm3	test_eager_padding_matches_padding_free_with_position_ids	AssertionError: Tensor-likes	N/A
NEW	minicpm3	test_sdpa_padding_matches_padding_free_with_position_ids	AssertionError: Tensor-likes	N/A
CHANGED	modernvbert	test_masked_lm_inference	RepositoryNotFoundError	RepositoryNotFoundError
NEW	moshi	test_eager_padding_matches_padding_free_with_position_ids	AssertionError: Tensor-likes	N/A
CHANGED	moshi	test_moshika_greedy_unconditional_fp16	AssertionError: np.False_	AssertionError
CHANGED	moshi	test_moshiko_greedy_unconditional_fp16	AssertionError: np.False_	AssertionError
CHANGED	moshi	test_moshiko_greedy_unconditional_fp32	AssertionError: np.False_	AssertionError
CHANGED	muse_glimmer	test_image_generation_matches_reference	OutOfMemoryError	OutOfMemoryError
CHANGED	muse_glimmer	test_text_generation_matches_reference	OutOfMemoryError	OutOfMemoryError
CHANGED	olmo	test_model_7b_logits	OutOfMemoryError	OutOfMemoryError
NEW	olmo	test_model_7b_twin_2t_logits	OutOfMemoryError	N/A
CHANGED	olmo3	test_generate_beyond_sliding_window	OutOfMemoryError	OutOfMemoryError
CHANGED	olmo3	test_real_model_7b_greedy_generation	OutOfMemoryError	OutOfMemoryError
CHANGED	pi0	test_train_pi0_base_libero	OutOfMemoryError	OutOfMemoryError
CHANGED	vision_encoder_decoder	test_forward_pass	RemoteEntryNotFoundError	RemoteEntryNotFoundError
CHANGED	vision_encoder_decoder	test_generation	RemoteEntryNotFoundError	RemoteEntryNotFoundError
CHANGED	wav2vec2_with_lm	test_save_load_pretrained_additional_features	FileExistsError	FileExistsError
CHANGED	wav2vec2_with_lm	test_save_load_pretrained_default	FileExistsError	FileExistsError

Quick analysis:

• OutOfMemoryError (25): Runner-to-runner memory variance across days. Same underlying OOM, different allocation numbers. Likely not regressions.
• AssertionError: np.False_ (5): mimi x2, moshi x3. Numpy changed how it represents booleans (False -> np.False_). Not a model regression. Prev error was generic AssertionError before the np.False_ repr appeared in the trace.
• AssertionError: Tensor-likes (3): minicpm3 x2, moshi x1. Appeared new in 08/14 single GPU. minicpm3::test_eager_padding was already in 08/13 multi GPU; the other two were not in 08/13 at all.
• AssertionError: Lists differ (1): gemma::test_model_7b_fp16_static_cache. Fixed by [Gemma] Update expected values for A10G #47976 (merged).
• AssertionError: string != (1): flava::test_inference. Diff got closer to expected (0.155 -> 0.0005), barely still fails 4-decimal tolerance.
• RepositoryNotFoundError (1): modernvbert. Repo access issue, pre-existing.
• RemoteEntryNotFoundError (2): vision_encoder_decoder. Remote file missing, pre-existing.
• FileExistsError (2): wav2vec2_with_lm. File conflict in test teardown, pre-existing.