Node external memory grows continuously on GridFS until OOM (heap stable) — RC 8.4.1 / Mongo 8.2.5 on K8s

[LinTechSo]

Hi,

I have issue that memory continuously grows until OOM on my RC
(heap stable, nodejs_external_memory_bytes never decreases) — 8.4.1

Description

I have a self-hosted Rocket.Chat instance where memory always grows over time until the pod is OOMKilled / restarts. This repeats in a cycle.

Below is the Rocket.Chat container memory usage over ~30 days. It shows a clear sawtooth: climb toward ~6 GiB → drop (restart/OOM) → climb again every ~7–10 days.

Debugging with metrics

To debug this, I enabled Rocket.Chat Prometheus metrics and monitored:

• nodejs_heap_size_total_bytes
• nodejs_heap_size_used_bytes
• nodejs_external_memory_bytes

Result:

• Heap used / heap total stay flat and healthy (~0.6–0.75 GiB)
• nodejs_external_memory_bytes grows continuously in a stair-step pattern and never comes down
• Container RSS follows the external memory line until OOM

So this is not a classic V8 JS heap leak. The growth is in Node external memory (Buffers / memory outside the V8 heap). am i wrong?

Debugging notes / current hypothesis

While debugging (including with LLM assistance), the evidence points to GridFS / file-related Buffer handling as the likely area:

• Storage type is GridFS
• Workspace has substantial file history and large binaries
• Heap stays flat while external grows and is not released after transfers
• A large external jump on one day did not match heavy uploads that same day (details below), which suggests download/preview/retention of Buffers — not only “big upload then free”

I expected that after upload/download finishes, allocated Buffer/external memory would be released (or at least stabilize). Instead it only grows until restart.

Server Setup Information

• Version of Rocket.Chat Server: 8.4.1 (rocketchat/rocket.chat:8.4.1)
• Deployment Method: Kubernetes (Helm), 1 replica
• MongoDB Version: 8.2.5 (replica set)
• Number of Running Instances: 1
• DB Replicaset Oplog / change streams: yes (replica set rs0)
• File storage: GridFS
• Pod memory limit (affected instance): 5734Mi

Upload corpus (Mongo rocketchat_uploads)

• Files: 42,140
• Total size: ~19.43 GiB
• Average size: ~0.47 MiB
• Max size: ~176.24 MiB

Important finding: Aug 5 external jump ≠ heavy uploads that day

On 2026-08-05, Grafana showed external memory jumping roughly ~1.9 GiB → 3.3 GiB (+1.4 GiB) and staying high.

But Mongo uploads that UTC day were small overall:

• Approx total uploaded that day: ~195 MiB
• Peak hour (UTC 15): ~70.8 MiB total, max file ~64.8 MiB
• Morning hours: mostly small clipboard/screenshot PNGs + thumbnails

So the large external step is not explained by that day’s upload volume. This supports looking at download/preview / retained Buffers on GridFS paths, not only upload.

Hourly upload summary for 2026-08-05 (UTC):

hourUTC	count	totalMB	maxMB
4	4	0.1	~0
5	18	2.6	0.9
6	32	4.8	0.9
7	39	5.4	1.0
8	49	35.5	26.6
9	71	12.0	1.1
10	44	4.2	0.5
11	60	11.5	2.1
12	48	16.2	12.7
13	20	10.9	8.6
14	25	15.4	3.5
15	17	70.8	64.8
16–22	low	<2	small

Expected behavior

Memory usage should stabilize. After file upload/download/preview completes, external/Buffer memory should be released (or reclaimable), not permanently accumulate until OOM.

Actual behavior

• Container memory grows continuously over days (see 30-day chart)
• nodejs_external_memory_bytes stair-steps upward and never decreases
• Heap remains stable
• Process recovers only after OOM/restart, then the cycle repeats
• Not proper stdout logs

Already checked

• Not a heap OOM (nodejs_heap_size_* flat)
• Increasing --max-old-space-size does not address external memory
• No documented setting/env found to cap/release Node external/Buffer memory for GridFS
• Upload buffering improvements from 8.1 are already in 8.4.1; 8.5.x notes did not show a clear fix for this continuous external growth pattern

Questions

1. Is unbounded growth of nodejs_external_memory_bytes with flat heap a known issue on GridFS (especially download/serve/preview paths)?
2. What is the recommended way to confirm Buffer retention on download/preview?
3. Aside from migrating off GridFS, is there any supported mitigation?

I would appreciate it if you could help me figure this out, and please correct me if there is any misunderstanding.

Thanks.

[RounakKumarAgarwal]

hi i would like to contribute @LinTechSo , can you assign it to me?

[LinTechSo]

Hi @RounakKumarAgarwal , thanks for your help and interest in contributing. Please feel free to investigate this issue and open a pull request referencing this issue. I can provide additional reproduction details or test results if needed. A maintainer can assign the issue to you if required i can not.

[RounakKumarAgarwal]

Hi @LinTechSo , thanks for the response. I'd like to investigate this issue and work on a fix. I'll start by reproducing the reported memory growth and tracing the external memory usage. If possible, could you provide the reproduction details/test scenario you mentioned? I'll reference #41734 in the PR once I have a fix

[reetp]

Rather than trust AI (the less you know the less you should trust it) it would be better if you simply read the docs on GridFS.

I have emphasised the relevant sections.

If you are using it then migrate your file store, clean the db, and test again. There at least two migrate scripts around.

By default, GridFS is used in Rocket.Chat for file storage because MongoDB offers this functionality with zero configuration

However, it is not recommended for production environments due to the high load it places on the database. GridFS can only be suitable if:

A retention policy is in place to manage file storage.

You are storing only a small number of lightweight files each week.

https://docs.rocket.chat/docs/recommendations-for-file-upload

[LinTechSo]

Thanks for the reply, and for the point about relying on AI alone. I’ll read through the GridFS documentation and keep your guidance in mind.

That said, I think it may still be worth considering improvements here. Since GridFS is the default and works without additional configuration, some deployments may depend on it—especially where introducing external S3-compatible storage or filesystem dependencies in HA envs is not straightforward.

I’d be interested in following and any contribution to work that improves this path.

[RounakKumarAgarwal]

Sure I'll do @LinTechSo