When I use the lmmes_eval repo to evaluate the benchmark, there is a option whether interleaves image and subtitles.
The lmms_eval make it false default, but I find some model evaluate it with subtitles, such as videochat_flash recently.
Should this be noted on the leaderboard?
When I use the lmmes_eval repo to evaluate the benchmark, there is a option whether interleaves image and subtitles.
The lmms_eval make it false default, but I find some model evaluate it with subtitles, such as videochat_flash recently.
Should this be noted on the leaderboard?