fix: remove # from ingress snippet

[vigneshrajsb]

Problem

nginx-controller consuming a lot of memory and OOMKills

Root Cause: # in PR value corrupts nginx-ingress cleanConf depth tracking

The nginx-ingress controller generates nginx.conf by rendering a Go template and then post-processing it through a function called cleanConf. This function re-indents the config by tracking brace depth — it increments depth on { and decrements on }, writing that many tabs at the start of each line.

Critical flaw: cleanConf treats # as a comment character in all contexts (it has no awareness of nginx string literals). Once it encounters #, it enters comment mode for the rest of that line — meaning any { or } characters after the # are ignored for depth tracking.

The ingressBannerSnippet function generated a configuration-snippet containing:

sub_filter "</head>" '<script>window.LFC_BANNER = [{"label":"uuid",...},{"label":"PR","value":"#20845","url":"..."},...];</script></head>';

On the single long line containing the JSON array, the sequence is:

...{"label":"PR","value":"  #  20845","url":"..."},{"label":"sha"...},{...},...];
                            ^
                     cleanConf enters comment mode here

When cleanConf hits # inside "value":"#20845", it stops counting braces for the rest of the line. The { that opened the PR object is counted, but its matching } and all subsequent {} pairs (sha, branch, service name, build) are not counted. This leaves the depth counter permanently +1 after each such location block is processed.

Impact

• 211 ingresses across lifecycle environments had this pattern (all PR-based deployments)
• Each ingress generates 2 server blocks (HTTP + HTTPS), each with a location containing the sub_filter
• ~422 depth increments accumulated → 431 levels of tab indentation at the end of the file
• The generated nginx.conf grew to 35MB (vs. a normal <1MB)
• During config reload, NGINX loads the new 35MB config alongside the old one in memory, causing transient spikes that breached the 1Gi limit and triggered OOMKills

This explains why the old cluster handled 500+ ingresses within 1Gi but this cluster could not — the old cluster did not have lifecycle environment ingresses with this #-in-value pattern at this scale.

Fix

Remove the # prefix from the PR value in the LFC_BANNER JSON:

- value: `#${pullRequestNumber}` || '',
+ value: `${pullRequestNumber}` || '',

The PR number without # is still meaningful since the full PR URL is already present in the url field. This eliminates the # character from the nginx snippet entirely, preventing cleanConf from entering comment mode mid-line.

Verification

After this change is deployed, the nginx.conf size should drop from ~35MB back to a normal range (<2MB), and memory usage per pod should stabilize well below 1Gi even during reloads.