Skip to content

Feature request: chunked row input and stream-friendly output for large server-side table exports (50k–100k rows) #1112

@Wesilnt

Description

@Wesilnt

Summary

We run a server-side PDF export service (Node.js, Docker, headless) that renders tables with jsPDF + jspdf-autotable . For large tables, peak memory usage is high enough to cause OOM in production, especially under concurrent exports.

We would like to discuss whether incremental/chunked row input and/or stream-oriented output are feasible, or what the recommended pattern is for very large tables.


Use case

Item Detail
Environment Node.js 20, Linux Docker (bookworm), container limit ~8 GB
Library stack jspdf@3.0.4, jspdf-autotable@5.0.8 (via @tms/pdf-common@0.3.53)
Typical load 1–3 tables per report; one table can be very large
Data fetch Backend API with pageSize up to 100,000 rows per request (single page)
Concurrency Up to ~30 concurrent export tasks (service limit); memory pressure multiplies

Representative failure scenario:
Pure table export, 3 widgets/tables, each with a large row count (e.g. tens of thousands of rows × dozens of columns). Process RSS grows until the container is killed (OOM).


Current pipeline (simplified)

→ fetch table data (pageSize = 100000) → ajaxData { rows, columns } in memory
→ TMSPdf / jsPDF-AutoTable → layout + full document in memory
→ doc.output('arraybuffer') → full PDF Buffer (second large allocation)
→ write PDF to disk / upload

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions