Add cache manager and package cache routing.

docs/proposals/unified-cache-manager.rst

@@ -0,0 +1,168 @@
+Proposal: Unified Cache Manager
+================================
+
+Problem
+-------
+
+Fromager's caching logic is scattered across multiple modules with no central
+coordination. Prebuilt wheels are checked separately from previously built
+wheels, remote wheel servers are not consulted during local builds, and there
+is no mechanism to share a base cache of common dependencies across
+hardware-specific bootstrap variants (e.g. CUDA, Gaudi). This leads to:
+
+- Redundant builds when wheels already exist in a remote cache or sibling
+  variant's output.
+- Duplicated storage of common (non-accelerated) dependencies in every
+  variant's output directory.
+- No visibility into cache hit rates, artifact integrity, or staleness.
+- No short-circuit path — even a full cache hit still downloads source and
+  sets up a build environment before discovering the wheel exists.
+
+Proposed Solution
+-----------------
+
+Introduce a unified ``CacheManager`` class that centralizes all cache
+operations behind a layered lookup strategy:
+
+Architecture
+~~~~~~~~~~~~
+
+.. code-block:: text
+
+   CacheManager
+   ├── CacheCollection("default")
+   │   ├── LocalDirectoryBackend(wheels-repo/downloads/)
+   │   ├── LocalDirectoryBackend(wheels-repo/prebuilt/)
+   │   └── RemotePEP503Backend(https://cache-server/simple/)  [optional]
+   └── CacheCollection("gaudi-ubi9")  [variant, if active]
+       └── LocalDirectoryBackend(wheels-repo-gaudi-ubi9/downloads/)
+
+Key components:
+
+- ``WheelCacheKey`` — Content-addresses artifacts by canonicalized package
+  name, version, and numeric build tag.
+- ``CacheBackend`` protocol — Abstract interface implemented by
+  ``LocalDirectoryBackend`` (filesystem) and ``RemotePEP503Backend``
+  (PEP 503 simple repository).
+- ``CacheCollection`` — Named group of backends searched in priority order
+  (e.g. "default" searches local then remote).
+- ``StoreRouter`` — Determines which collection owns a given package based on
+  the variant's top-level requirements file and optional overrides.
+- ``CacheManager`` — Orchestrates hierarchical lookup across collections with
+  fallback from variant to default.
+
+Lookup and store routing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+On lookup, the manager searches the active variant collection first, then
+falls back to the default collection. Within each collection, backends are
+queried in registration order (local before remote).
+
+On store, the ``StoreRouter`` routes packages explicitly listed in the
+variant's ``requirements.txt`` to the variant collection directory. All
+unlisted transitive dependencies are stored in the default collection. This
+prevents duplication of common packages across variant outputs.
+
+Short-circuit optimization
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When a cache hit is found during the ``PREPARE_SOURCE`` phase, the
+bootstrapper skips source download, build environment creation, and build
+dependency resolution entirely — proceeding directly to install dependency
+extraction from the cached wheel's metadata. This eliminates the most
+expensive steps for packages that do not need rebuilding.
+
+Remote cache with integrity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``RemotePEP503Backend`` lazily fetches per-project package indices on first
+access and maintains a session-scoped in-memory index. Downloads are verified
+with streaming SHA256 checksums, use atomic temporary files, and reject
+plaintext HTTP URLs that lack integrity hashes (unless ``--cache-allow-insecure``
+is passed for development workflows). Filenames are sanitized to prevent path
+traversal attacks.
+
+Observability
+~~~~~~~~~~~~~
+
+A new ``fromager cache`` CLI command group provides:
+
+- ``cache list`` — Show all cached artifacts with versions and build tags.
+- ``cache stats`` — Display hit/miss counts and rates from the last run.
+- ``cache verify`` — Validate integrity of local cache contents.
+- ``cache invalidate`` — Remove specific artifacts by name/version/tag.
+- ``cache gc`` — Garbage-collect old build tags, keeping only the N most
+  recent per package+version.
+
+Scope
+-----
+
+The new cache subsystem is opt-in via ``--use-cache-manager`` on the
+``bootstrap`` command. When disabled, existing behavior is preserved
+unchanged.
+
+Files added or modified:
+
+- ``src/fromager/cache.py`` — New module with all cache classes.
+- ``src/fromager/commands/cache_cmd.py`` — CLI commands and factory function.
+- ``src/fromager/commands/bootstrap.py`` — Wiring ``--use-cache-manager`` and
+  ``--cache-allow-insecure`` options.
+- ``src/fromager/bootstrapper.py`` — Short-circuit path and store routing for
+  built wheels.
+- ``src/fromager/context.py`` — ``cache`` property on ``WorkContext``.
+- ``src/fromager/requirements_file.py`` — ``CACHED`` source type.
+- ``tests/test_cache.py`` — Unit tests for the cache subsystem.
+- ``tests/test_bootstrapper_iterative.py`` — Integration tests for cache
+  dispatch and store routing.
+
+Benefits
+--------
+
+- Eliminates redundant builds when wheels exist in a remote or sibling cache.
+- Reduces bootstrap time by short-circuiting cached packages (skips source
+  download, build env setup, and build dep resolution).
+- Prevents duplication of common dependencies across variant outputs via
+  hierarchical store routing.
+- Provides cache observability through dedicated CLI commands.
+- Enforces artifact integrity with SHA256 verification and atomic writes.
+- Extensible to future metadata (``fromager.json`` in ``.dist-info/``) for
+  build dependency tracking and accelerator annotations.
+
+Security considerations
+-----------------------
+
+- Remote downloads are verified against SHA256 hashes declared in PEP 503
+  index pages. Mismatched files are deleted and raise an error.
+- Plaintext HTTP URLs without SHA256 hashes are rejected by default.
+  The ``--cache-allow-insecure`` flag explicitly opts in for internal or
+  development registries.
+- Filenames from remote indices are sanitized to prevent directory traversal.
+- Local cache writes use atomic ``tempfile`` + ``rename`` to prevent readers
+  from observing partial files.
+- ``scan()`` skips symlinked wheels to prevent ``invalidate``/``gc`` from
+  deleting files outside the cache root.
+- Fetch failures (network errors, hash mismatches) are caught and treated as
+  cache misses, falling through to the next backend or a fresh build.
+
+Verification
+------------
+
+- All existing unit and e2e tests pass unchanged (legacy path preserved).
+- 169 new tests cover cache components, short-circuit logic, store routing,
+  and error handling.
+- Validated on walkerpass4 with both local and remote caches:
+  pre-populated local cache achieves full hit rate; remote PEP 503 server
+  correctly populates local cache and only uncached packages trigger builds.
+- Linting (``ruff``), type checking (``mypy``), and formatting all pass.
+
+Future work
+-----------
+
+- ``fromager.json`` in ``.dist-info/`` to record build dependencies,
+  accelerator annotations, and provenance for cached wheels.
+- Integration with ``build_tag_hook`` (issue #1059) for platform-suffixed
+  cache keys.
+- Automatic detection of accelerated packages via ELF inspection or wheel
+  tag analysis.
+- Promotion of ``--use-cache-manager`` to default behavior once proven in
+  production.

pyproject.toml

@@ -128,6 +128,7 @@ canonicalize = "fromager.commands.canonicalize:canonicalize"
 download-sequence = "fromager.commands.download_sequence:download_sequence"
 wheel-server = "fromager.commands.server:wheel_server"
 lint-requirements = "fromager.commands.lint_requirements:lint_requirements"
+cache = "fromager.commands.cache_cmd:cache"
 
 [tool.coverage.run]
 branch = true

src/fromager/bootstrapper.py

@@ -325,23 +325,52 @@ def _download_wheel_from_cache(
         return None, None
 
 
+def _find_cached_wheel_via_manager(
+    ctx: context.WorkContext,
+    req: Requirement,
+    resolved_version: Version,
+) -> tuple[pathlib.Path | None, pathlib.Path | None]:
+    """Cache lookup using the CacheManager (thread-safe, no Bootstrapper state).
+
+    Delegates to ``ctx.cache.lookup_wheel()`` for a unified hierarchical
+    lookup across collections.
+
+    Returns:
+        Tuple of (cached_wheel_filename, parent_dir_as_sentinel).
+        Both None if no cache hit. The second value is non-None on hit
+        to signal the caller that a cached wheel was found.
+    """
+    assert ctx.cache is not None
+    pbi = ctx.package_build_info(req)
+    build_tag = pbi.build_tag(resolved_version)
+
+    result = ctx.cache.lookup_wheel(req, resolved_version, build_tag)
+    if not result.hit:
+        return None, None
+
+    assert result.path is not None
+    return result.path, result.path.parent
+
+
 def _find_cached_wheel(
     ctx: context.WorkContext,
     cache_wheel_server_url: str | None,
     req: Requirement,
     resolved_version: Version,
 ) -> tuple[pathlib.Path | None, pathlib.Path | None]:
-    """Look for cached wheel in 3 locations (thread-safe, no Bootstrapper state).
+    """Look for cached wheel (thread-safe, no Bootstrapper state).
 
-    Checks for cached wheels in order:
-    1. wheels_build directory (previously built)
-    2. wheels_downloads directory (previously downloaded)
-    3. Cache server (remote cache)
+    When a CacheManager is configured on the context, delegates to it
+    for a unified hierarchical lookup across collections. Otherwise
+    falls back to the legacy per-directory search.
 
     Returns:
         Tuple of (cached_wheel_filename, unpacked_cached_wheel).
         Both None if no cache hit.
     """
+    if ctx.cache is not None:
+        return _find_cached_wheel_via_manager(ctx, req, resolved_version)
+
     cached_wheel, unpacked = _look_for_existing_wheel(
         ctx, req, resolved_version, ctx.wheels_build
     )
@@ -1753,6 +1782,33 @@ def _phase_prepare_source(self, item: WorkItem) -> list[WorkItem]:
         sdist_root_dir = prepared.sdist_root_dir
         item.cached_wheel_filename = prepared.cached_wheel_filename
 
+        # Short-circuit: when CacheManager provides a hit, skip directly to
+        # PROCESS_INSTALL_DEPS -- no source download, no build env, no build
+        # deps resolution needed. Install deps are extracted from the wheel.
+        if prepared.cached_wheel_filename and self.ctx.cache is not None:
+            pbi = self.ctx.package_build_info(item.req)
+            build_tag = pbi.build_tag(item.resolved_version)
+            self.ctx.cache.store_wheel(
+                item.req,
+                item.resolved_version,
+                build_tag,
+                prepared.cached_wheel_filename,
+            )
+            server.update_wheel_mirror(self.ctx)
+            unpack_dir = _create_unpack_dir(
+                self.ctx.work_dir, item.req, item.resolved_version
+            )
+            item.build_result = SourceBuildResult(
+                wheel_filename=prepared.cached_wheel_filename,
+                sdist_filename=None,
+                unpack_dir=unpack_dir,
+                sdist_root_dir=None,
+                build_env=None,
+                source_type=SourceType.CACHED,
+            )
+            item.phase = BootstrapPhase.PROCESS_INSTALL_DEPS
+            return [item]
+
         assert sdist_root_dir is not None
 
         if sdist_root_dir.parent.parent != self.ctx.work_dir:
@@ -1990,6 +2046,20 @@ def _phase_build(self, item: WorkItem) -> list[WorkItem]:
             cached_wheel_filename=item.cached_wheel_filename,
         )
 
+        # Route newly built wheels to the appropriate collection directory.
+        # This copies the wheel into the routed collection's storage while
+        # keeping the original in downloads/ for the internal wheel server.
+        if (
+            wheel_filename is not None
+            and self.ctx.cache is not None
+            and not item.cached_wheel_filename
+        ):
+            pbi = self.ctx.package_build_info(item.req)
+            build_tag = pbi.build_tag(item.resolved_version)
+            self.ctx.cache.store_wheel(
+                item.req, item.resolved_version, build_tag, wheel_filename
+            )
+
         source_type = sources.get_source_type(self.ctx, item.req)
 
         item.build_result = SourceBuildResult(