Parser memory reduction

[xeioex]

This series substantially reduces memory consumption and compilation time in the built-in njs engine. It introduces a compact representation for array literals, reduces the size of common parser structures, and separates temporary parser and generator state from data that must remain alive after compilation.

Previously, each element in an array literal was lowered through the generic object-property path. A single element such as x in [x, x, ...] required six parser nodes, several generator continuation entries, an interned numeric property key, and a generic property-initialization instruction. At one million elements, compilation retained more than 1 GB in the VM memory pool.

Array elements are now represented by compact { value, index } records and generated sequentially using a dedicated immediate-index initialization instruction. This removes five parser nodes per element, avoids numeric index constants, keeps generator continuation depth constant, and reduces generated bytecode from approximately 32 MB to 24 MB.

The series also fixes an existing correctness issue: array literals larger than 32,760 elements previously had the expected length but did not contain their initialized values after switching to the slow-array representation.

Structure Changes

Structure	Before	After	Pool allocation class
njs_parser_node_t	104 B	64 B	128 B to 64 B
njs_variable_reference_t	40 B	16 B	embedded
njs_parser_scope_t	136 B	104 B	256 B to 128 B
njs_variable_t	96 B	56 B	128 B to 64 B

Active labels are stored in a parser-wide indexed set instead of allocating complete variables and wrappers in every scope. Statement labels use atom identifiers instead of storing an njs_str_t in every parser node.

Compilation Lifetime

Each parser invocation now owns a temporary memory pool. It contains parser nodes, compact array-item storage, parser continuations, lexer preread tokens, active labels, generator stack entries and copied contexts, generator blocks and jump patches, traversal and index-cache scratch, non-global scopes, local variables and variable-tree nodes, reference maps, declaration arrays, and temporary closure-index builders.

The pool is destroyed after bytecode generation and on parser or generator failure in the main script, module, and runtime Function compilation paths.

The global scope and global variables, atoms and parser-created strings, function lambdas and regexp patterns, generated bytecode and line maps, runtime constants, module records, and finalized closure-index arrays remain persistent because runtime state references them.

Closure indexes are accumulated in temporary arrays and copied into exact-sized VM-owned storage before the compilation pool is destroyed. Persistent global variables retain declaration-origin type information without retaining pointers to released block scopes.

The series also adds local failure cleanup for AST serialization and provisional module compilation. Failed module compilation removes the provisional module and releases generated code records, line maps, and code buffers created after the compilation checkpoint. Top-level global compilation is intentionally not made transactional; callers should discard the VM after a failed top-level compilation.

Results

Median fresh-process measurements on AArch64:

Workload	Baseline RSS	Current RSS	Reduction	Baseline time	Current time
Array references 1M	1,097,232 KiB	156,180 KiB	85.8%	3.96 s	0.57 s
Declarations 100K	146,168 KiB	104,072 KiB	28.8%	0.41 s	0.25 s
Labels 100K	76,064 KiB	47,884 KiB	37.1%	0.10 s	0.06 s
test/buffer.t.js	10,740 KiB	9,400 KiB	12.5%	<0.01 s	<0.01 s

Retained VM memory after compilation:

Workload	Baseline	Current	Reduction
Array references 1M	1,043.4 MB	34.2 MB	96.7%
Declarations 100K	127.7 MB	57.3 MB	55.1%
Labels 100K	47.7 MB	7.7 MB	83.9%
test/buffer.t.js	2.77 MB	0.59 MB	78.8%

The semantic-state lifetime changes provide additional retained-memory reductions relative to the initial node-only temporary pool:

Workload	Node-only pool	Final	Additional reduction
100K sequential block declarations	39.1 MB	7.1 MB	81.9%
20K arrow closures	17.2 MB	8.6 MB	50.2%
20K declared functions with closures	37.0 MB	17.5 MB	52.6%

Peak RSS changes little during the semantic-state stage because scopes, variables, references, and AST nodes are all needed until generation completes. The primary benefit is reducing memory retained by long-lived VMs.

NGINX Master and Workers

A separate benchmark loaded a 6.89 MB module containing a one-million-element numeric array through js_import, using the njs engine and four workers.

Metric	Baseline	Current	Reduction
Master RSS before requests	1,116.7 MiB	172.9 MiB	84.5%
Worker RSS before requests	1,117.3 MiB	173.7 MiB	84.5%
Process-group PSS before requests	1,118.5 MiB	175.0 MiB	84.4%
Process-group PSS after warmup	1,248.3 MiB	399.3 MiB	68.0%

The imported compilation state is initially almost entirely shared through copy-on-write. After all workers execute the module, each worker materializes private runtime array state, but the process group remains approximately 849 MiB smaller than the baseline. Memory remained stable after 100, 1,000, and 10,000 requests and after a subsequent idle period.

NGINX startup time, including configuration loading, import compilation, daemon startup, and worker creation, improved from approximately 4.43 seconds to 0.71 seconds.

Stress Benchmarks

The benchmark suite now includes focused generated workloads for one million array references, 100,000 sequential block declarations, 100,000 sequential labels, 100,000 unique declarations, 100,000 referenced declarations, 20,000 closures, and 10,000 nested labels.

These are scaling stress tests rather than isolated parser timings. Their measured interval includes source construction, runtime Function compilation, VM cloning, and cleanup.

Validation

• Value-preservation analysis is limited to 128 recursive levels. Deeper expressions conservatively request a temporary copy instead of risking native stack exhaustion.
• Disassembly comparisons against the parent commit show identical opcode histograms and MOVE counts for bench4.js, raytrace.js, deltablue.js, navier-stokes.js, splay.js, richards.js, and crypto.js. The depth guard therefore adds no bytecode overhead for these ordinary workloads.

[VadimZhestikov]

Looks good

Notes (non-blocking)

1. The two bug fixes (large-array data loss, stack-exhaustion DoS) deserve CHANGES entries; the DoS is remotely
   triggerable via crafted source and is a backport candidate.
2. I verified the pool-lifetime and module-cleanup logic by reading plus the ASAN suite; the failure paths for module
   cleanup aren't exercised by a specific unit test I could see (they'd need a module that fails to compile after
   provisional publish), but the logic is straightforward and defensive.
3. The stress benchmarks (commit db592c9) are infrastructure to measure the memory/scaling win — I didn't benchmark,
   as it's not a correctness concern.

[xeioex]

the DoS is remotely triggerable via crafted source and is a backport candidate

JS code is trusted according to NJS thread model.