Fix NDI consumer memory leak

[leonedo]

Problem Description

Fix for issue#1591

Since CasparCG 2.4, the NDI consumer has exhibited a memory leak, particularly when used with some integrated graphics (like Intel Iris Xe). The issue can also affect Nvidia discrete GPU users. The leak occurs upon starting NDI output and causes system memory to continuously grow until the system runs out of RAM (or VRAM for discrete GPUs).

Root Cause
The NDI consumer's frame buffer queue had no size limit, allowing frames to accumulate indefinitely when the NDI send operation cannot keep up with the incoming frame rate. On Intel integrated graphics (which share system RAM), this manifests as an obvious memory leak. On Nvidia discrete GPUs (with dedicated VRAM), the issue is less noticeable.
Additionally, frame objects were held in memory longer than necessary, not being released until the next loop iteration after the sleep period.

Proposed Solution
This fix implements three key changes:

1. Buffer Size Limiting
   Limits the frame buffer to a maximum of 8 frames. When the buffer exceeds this limit, excess frames are dropped rather than accumulated. This prevents unbounded memory growth while maintaining smooth output.

while (buffer_.size() > 8) {
    buffer_.pop();
    drop_count++;
}

2. Smart Logging
   Aggregates dropped frame warnings to log once per second rather than per-frame, preventing log spam while still providing visibility into system performance.

if (drop_count > 0 && elapsed >= 1 second) {
    CASPAR_LOG(warning) << L"NDI dropped " << drop_count << L" frames in last second";
    drop_count = 0;
}

3. Prompt Frame Cleanup
   Explicitly releases frame memory after sending and sleeping, rather than waiting for the next loop iteration. This ensures immediate memory reclamation.

std::this_thread::sleep_until(time_point);
frame = core::const_frame();  // Release frame memory

Performance Notes
On systems where NDI encoding cannot maintain real-time performance (particularly Intel integrated graphics at high resolutions/framerates), this fix will drop frames to maintain system stability rather than accumulating them in memory. This is the correct behavior - graceful degradation is preferable to memory exhaustion.
Users with high-performance discrete GPUs should see no change in behavior, as the buffer limit of 4 frames is sufficient for normal timing jitter absorption.
Files Changed

modules/newtek/consumer/newtek_ndi_consumer.cpp

Future Enhancements (Optional)
The buffer size could potentially be made configurable via the CasparCG configuration file to allow power users to tune performance for their specific hardware:

<ndi>
  <name>Output Name</name>
  <buffer-size>4</buffer-size>
</ndi>

However, this is not included in the current fix to keep the change minimal and focused on resolving the memory leak.

[dimitry-ishenko]

I don't use NDI and I am not familiar with its inner workings, but what about a scenario where you have spurious congestions on the network and frames arrive in bunches (eg, 10 at a time)? They would be unnecessarily dropped, whereas in the current scenario they would get played out after a slightly larger delay.

Perhaps bigger default buffer value is the answer? And people who want near real-time playback can dial it down.

[leonedo]

I see your point, these are going out, but I guess NDI's congestion control can cause send_send_video_v2() to block during network congestion. In that scenario, frames would accumulate in the buffer while waiting for network capacity.

With buffer=8, temporary network congestion might cause frame drops even though the frames could be successfully sent if we waited longer.

I'm leaning toward 16 frames by default to handle the scenarios you described, any thoughts?

The critical fix is preventing unbounded growth (∞ frames → memory leak). Any reasonable limit (12/16/32) is better than unlimited.

And I think we can also add buffer-size in the config file and let users change it if needed

[dimitry-ishenko]

I'm leaning toward 16 frames by default to handle the scenarios you described, any thoughts?

Sorry, I am not familiar with the innards of NDI to have any thoughts on the subject 😅 Ideally, someone who's using NDI can chime in.