Qwen3-VL video processor spends the full pixel budget on short clips, making a 90s clip cost nearly as many tokens as a 10-minute film

[dkrisman]

System Info

• Mechanism verified against current main: src/transformers/models/qwen3_vl/video_processing_qwen3_vl.py (smart_resize, Qwen3VLVideoProcessor defaults fps = 2, max_frames = 768)
• Measurements below taken with the transformers bundled in vLLM v0.27.1 serving Qwen3.8-27B-FP8; the relevant code is unchanged on main

Who can help?

@zucchini-nlp (video processors)

Reproduction

smart_resize(num_frames, height, width, ..., max_pixels) treats max_pixels as the total t*h*w budget across all sampled frames: when t_bar * h_bar * w_bar > max_pixels every frame is scaled down by the same factor. Frame sampling is fps=2 capped at max_frames=768, so the per-frame share of the budget is inversely proportional to clip length:

• a 10-minute 1080p film samples 768 frames and each frame gets max_pixels / 768
• a 90-second 1080p clip samples 180 frames and each frame gets max_pixels / 180, i.e. 4x the resolution of the film's frames

Measured with the model card's long-video setting (size.longest_edge = 469762048, about 224K video tokens per item) on 1080p H.264 sources:

input	frames sampled	video embeds	tokens/frame
90 s clip	180	183,600	~1,020
10 min film	768	221,184	~288

The short clip costs 83% as much context as the full film while carrying 15% of its duration. The CPU-side resize cost scales the same way (minutes of preprocessing for the short clip at near-native resolution).

Expected behavior

A way to make token cost roughly proportional to clip duration. The current behavior may well be intentional (fewer frames leaves budget for per-frame detail), so rather than changing the default we would propose an opt-in video processor kwarg, e.g. proportional_frame_budget=False by default, which when enabled caps the effective budget at num_frames * (max_pixels // max_frames) so every clip gets the same per-frame pixel share a full-length max_frames video would. Clips sampling >= max_frames frames are unaffected by construction.

We run exactly that cap as a local patch in a serving deployment: the 90 s clip drops from 183,600 to ~53K tokens, end-to-end request time drops ~30%, and answer quality on the clip was unchanged in our (subjective, non-benchmark) checks.

One implementation caution from deploying that patch: some callers invoke smart_resize with tiny synthetic clips (vLLM's boot-time profiling probes with 2-frame dummy videos to size its encoder cache). A cap keyed on num_frames collapses those probes and, downstream, broke every video request until we exempted small frame counts (num_frames >= 32 guard). If this lands inside the processor the same consideration applies.

Happy to send a PR for the opt-in kwarg if maintainers are open to it.

[jyothimalla]

Hi @dkrisman,
Thanks for the detailed writeup and the concrete before/after numbers — very clear. I'd like to pick this up. Planning to implement the opt-in proportional_frame_budget kwarg as you proposed, including the num_frames >= 32 guard for tiny synthetic clips (good catch on the vLLM boot-probe case). @zucchini-nlp, let me know if this approach looks right before I open a PR.

[dkrisman]

https://github.com/dkrisman/transformers/tree/qwen3vl-video-max-pixels-per-frame has my fix for this issue. Feel free to use as much of my patch as you want or I am happy to open a PR from that branch.

[jyothimalla]

Thanks @dkrisman, that's a clean implementation and saves us duplicating the work. I like max_pixels_per_frame as an explicit kwarg better than a boolean flag — it's more flexible and avoids baking in the num_frames * (max_pixels // max_frames) formula.

One thing I noticed: the branch doesn't include the num_frames >= 32 guard you mentioned for vLLM's boot-time 2-frame dummy probes. Since it's opt-in (None by default) it's low-risk as-is, but if max_pixels_per_frame ever gets set as a processor default rather than passed per-call, tiny synthetic clips could still get collapsed. Would you mind adding that guard (or something equivalent) before opening the PR? Happy to review once it's up — go ahead and open it from your branch whenever you're ready.