Eliminate cache tarballs by synthesizing packuments at build time

[ogajduse]

Summary

• Instead of generating and storing a pre-built npm cache tarball as an extra Source, synthesize minimal registry packuments from source tarballs during %prep
• npm install --offline resolves the full dependency tree without any external artifact
• Removes generate_npm_tarball.sh (dead code), createNpmCacheTar(), and superfluous %clean section
• Net code reduction: 4 files changed, +86/-73 lines

Background

foreman-packaging stores bundle-strategy npm cache tarballs directly in git as regular blobs (not git-annexed), because those tarballs have no upstream URL to fetch them from. This adds up to 113 MB across 43 packages currently, and grows every time a bundled package is regenerated. Full writeup: theforeman/foreman-packaging#13080 (comment)... (see also the discussion on ogajduse#1).

How it works

No tarball is generated at all - not as a static asset, not at build time. Instead:

1. npm cache add puts each source tarball's content into _cacache/content-v2/ (npm does this natively)
2. An inline Node.js script in %prep extracts package.json from each tarball, computes integrity hashes (sha512/sha1), and writes ~2KB JSON packument entries directly into npm's _cacache/index-v5/ and _cacache/content-v2/ stores - this is the registry metadata npm needs to map package names/versions to the already-cached tarball content
3. npm install --offline finds both tarball content and registry metadata in the cache and resolves/installs successfully

Old flow: npm install online -> capture _cacache -> tar it -> store in git -> extract at build time
New flow: npm cache add each Source + synthesize packument JSON -> cache is ready in-place

Bugs found and fixed during prototyping:

• Scoped package cache keys need %2f encoding (e.g. @babel%2fruntime-corejs2), not a literal /
• Some tarballs use non-standard internal paths, so package.json is located via tar tf listing instead of assuming package/package.json
• Packages with multiple version tarballs in the same build (e.g. warning@3.0.0 + warning@4.0.3) need all versions accumulated into one packument, not overwritten

Build validation

Tested with nodejs-patternfly-react 2.40.0 (92 dependencies, including scoped packages and multi-version deps):

Phase	Target	Build ID	Result
Container test	react-dom (7 deps)	—	PASS
Container test	patternfly-react (92 deps)	—	PASS
rpmbuild -bp	Fedora 43 local	—	PASS
Copr build	fedora-rawhide-x86_64	10660261	PASS
obal scratch	rhel-9-x86_64	10660318	PASS
Release bump rebuild	rhel-9-x86_64	10660491	PASS
Foreman e2e build (depends on our RPM)	rhel-9-x86_64	10660510	PASS

This affects only the bundle strategy (single.mustache is untouched). foreman-packaging currently has 42 bundle-strategy nodejs packages; so far only nodejs-patternfly-react has been tested end-to-end with this change.

Impact on foreman-packaging

Eliminates cache tarballs entirely once packages are regenerated with this template - no external hosting infrastructure (Pulp, S3, HTTP server, etc.) needed to solve the storage problem.

Requirements

• %{sources} RPM macro (rpm >= 4.19): available on Fedora rawhide and RHEL 9+. Not available on RHEL 8.
• Node.js and npm >= 7 at build time (already BuildRequires)

Open question: macro extraction

The synthesize script (~80 lines) is fully generic - no package-specific logic, just takes a cache dir and tarball list. It's currently inlined into bundle.mustache, which adds ~80 lines to every generated spec but requires no extra BuildRequires and has no build-ordering constraints.

We discussed extracting it into an RPM macro on ogajduse#1. Considered homes:

• nodejs-packaging (Fedora-owned) - would need Fedora nodejs SIG buy-in
• foreman-build subpackage of foreman.spec - rejected, since it would force all ~42 nodejs bundle packages to wait for foreman to build first (currently they build in parallel)
• A new npm2rpm-macros package/subpackage - npm2rpm already owns this logic, would be a tiny package with no runtime dependencies

Proposing to ship inline for this PR and revisit macro extraction as a follow-up once this is validated across more of the 42 bundle packages in foreman-packaging.

Test plan

• All 17 existing unit tests pass
• Copr build on fedora-rawhide-x86_64
• obal scratch build on rhel-9-x86_64
• Foreman e2e build against resulting RPM

🤖 Generated with Claude Code

[ekohl]

npm install --offline resolves the full dependency tree without any external artifact

Is this new? Back when I built this solution I couldn't find anything that would do this, but to be fair, that was before npm install --offline existed.

nodejs-packaging (Fedora-owned) - would need Fedora nodejs SIG buy-in

And it would be a while before we can use this on RHEL.

Proposing to ship inline for this PR and revisit macro extraction as a follow-up once this is validated across more of the 42 bundle packages in foreman-packaging.

I think this is pretty good in the short term. In foreman-packaging we already have logic to regenerate the entire spec file so updating it after the fact is easier.

[ekohl]

This solves a massive issue I always had with bundled dependencies. Really neat.

[ekohl]

Because the %prep phase now uses node directly, should this be an unconditional dependency?

[ekohl]

I've spent quite a bit of time playing with this and reading the npm sources. This is effectively inlining what the cacache package does. That in turn uses ssri.

The hardcoding of content-v2 and index-v5 means it's tightly coupled to those packages. In the short term, this will probably be OK but longer term it can break.

The synthesize script (~80 lines) is fully generic - no package-specific logic, just takes a cache dir and tarball list. It's currently inlined into bundle.mustache, which adds ~80 lines to every generated spec but requires no extra BuildRequires and has no build-ordering constraints.

There's also another solution to this: create a Source: entry for the file. If you store it in git itself, it should be deduplicated internally. You can also put it on the internet somewhere so regular spectool can fetch it, and we could even annex it. Given its size, I wonder how useful that is.

The ideal was if npm itself actually implemented this. So a variation on option 1: patch npm (ideally in packaging) to synthesize a registry cache. Perhaps the Fedora / RHEL NPM maintainers are open to that. I'd be happy to help start that conversation.

[ekohl]

Now that it doesn't have to be filtered, I think you can call it in a single command

Suggested change

	for tgz in %{sources}; do
	echo $tgz | grep -q registry.npmjs.org || npm cache add --cache %{npm_cache_dir} $tgz
	npm cache add --cache %{npm_cache_dir} $tgz
	done
	npm cache add --cache %{npm_cache_dir} %{sources}

[ekohl]

If you could use libraries, I think this would be equivalent to:

for (const [name, packument] of Object.entries(packuments)) {
  const packBuf = Buffer.from(JSON.stringify(packument));

  const url = 'https://registry.npmjs.org/' + name.replace('/', '%2f');
  const cacheKey = 'make-fetch-happen:request-cache:' + url;
  const metadata = {
    time: 0,
    url: url,
    reqHeaders: {accept: 'application/json'},
    resHeaders: {'cache-control': 'public, max-age=300', 'content-type': 'application/json'},
    options: {}
  }

  cacache.put(cacheDir, cacheKey, packBuf, { metadata: metadata });
}

[ekohl]

I think this is cleaner:

Suggested change

	const encodedName = name.replace('/', '%2f');
	const cacheKey = 'make-fetch-happen:request-cache:https://registry.npmjs.org/' + encodedName;
	const url = 'https://registry.npmjs.org/' + name.replace('/', '%2f');
	const cacheKey = 'make-fetch-happen:request-cache:' + url;

[ekohl]

Assuming you defined url above:

Suggested change

	url: 'https://registry.npmjs.org/' + encodedName,
	url: url,

[ekohl]

I've created theforeman/foreman-packaging#13702 as a standalone PR to make it easy to bulk rewrite all packages.