Update spack-stack-dev from spack develop 2026/01/09 #574
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Due to performance and modeling reasons, the specs that are
reused in concretization are represented in the ASP problem
with their pure build dependencies trimmed.
On `develop` this means that any preference of the form:
```yaml
packages:
c:
prefer:
- llvm
```
is "disregarded" for purely reused specs, since the solver
is not aware of the compiler nodes used for them.
This commit fixes the semantics of virtual preferences so
that they are respected also for reused specs. It does so
without representing their compilers as nodes - so the modeling
of the DAG is not steered towards different optimal solutions
by the missing compilers.
Signed-off-by: Massimiliano Culpo <[email protected]>
* Handle URL paths Signed-off-by: Ryan Krattiger <[email protected]>
The current condition under which Spack starts a POSIX jobserver is whether all `gmake`s and `ninja`s in the DAG support `--jobserver-auth=FIFO:PATH`. That's does not take into account the (currently) unmodelled implicit dependency of `gcc` on `gmake`. GCC's `lto-wrapper` executable generates a temporary Makefile and invokes make either from the `MAKE` environment variable, or by looking in `PATH`. I don't think this is very trivial to fix, other than making GCC inject `gmake` as a build dependency, but that leads to circular dependencies since `gmake` is used to build GCC. Easier to revert for now and move this feature to the experimental installer PR where we can probably get away with it for now without a fix. This reverts commit 9c1ed1a. Signed-off-by: Harmen Stoppels <[email protected]>
…ases (spack#50477) * Append to install-time test log for multiple test phases Signed-off-by: Alex Richert <[email protected]> * log.py: fix tmp dir arg Signed-off-by: Alex Richert <[email protected]> * log.py: fix typo (:) Signed-off-by: Alex Richert <[email protected]> * use existing tests/pkg Signed-off-by: Alex Richert <[email protected]> * remove new test file/pkg Signed-off-by: Alex Richert <[email protected]> * test/cmd/install.py: skip log append check for win32 Signed-off-by: Alex Richert <[email protected]> * test/cmd/install.py: skip install 'tests' for win32 Signed-off-by: Alex Richert <[email protected]> * test/cmd/install.py: fix install test-skip logic (win32) Signed-off-by: Alex Richert <[email protected]> * test/cmd/install.py: skip test_package_output for windows Signed-off-by: Alex Richert <[email protected]> * Revert "test/cmd/install.py: skip test_package_output for windows" This reverts commit f14beed. Signed-off-by: Alex Richert <[email protected]> --------- Signed-off-by: Alex Richert <[email protected]>
Third try to enable concretization caching in `develop`. Past attempts were reverted due to cache management issues and race conditions. See spack#48198 for the introduction of the concretization cache, manifest, etc. That PR only added testing for the cache itself, not the manifest or cleanup procedures. This PR: - [x] Adds tests for cleanup - [x] Refactors the cache to store base32 hashes like the rest of Spack's hashes - [x] gzips cache entries for a smaller memory footprint - [x] Removes redundant stat calls - [x] Less aggressive/error prone cleanup phase - [x] Adds an entry limit to the cache, removes the size limit - [x] Rewrite pruning to use LRU instead of FIFO - [x] Canonicalizes and strips `clingo` input before hashing, for greater hit rate This PR simplifies the locking around the concretization cache. Entries are stored by the base32 hash of their inputs, and there is a single lock file (as we do in many places in spack) for all such entries. We use byte range locks to manage all the entries. Cleanup has changed to be less expensive overall. There is no more size limit; there is only an entry limit, and specs in the cache are compressed. Specs are ~35k, and the initial limit is 1000 cache entries, or ~35 MB. This limit is customizable in configuration. To check if the cache needs cleanup, we need only list files in the cache directory. To clean up, we stat the files and clean up the least recently used concretizations. We remove half the cache at a time so that we do not pay for cleanup on every concretization when we're close to the cache limit. The concretization cache is stored in the `misc_cache` by default, so `spack clean -m` is all that is needed to get rid of it. The location of the concretization cache can be specified independently (with a `url` config option) but we will document and expand on that use later as we integrate it in CI. --------- Signed-off-by: John Parent <[email protected]> Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Todd Gamblin <[email protected]>
…ack#51118) This commit fundamentally changes how externals are handled in the solver and introduces new methods for defining dependencies between them, addressing the design described in issue spack#49697. Core Solver Change: ------------------- Previously, externals were treated as a form of "configuration" and handled as a pervasive special case throughout the solver rules. With this change, externals are now modeled as fully concrete, installed specs *before* the solve begins. This allows the solver to treat them just like any other installed package, reusing them as available. This greatly simplifies the solver logic and removes numerous special cases. This new model enables more robust dependency specification, which is provided in two new formats: Inline Spec Syntax: ------------------- Users can now specify dependencies directly in the main `spec:` string, leveraging the familiar Spack syntax. packages: mpileaks: externals: - spec: "[email protected] %mpich %callpath" prefix: /user/path Spack resolves these dependencies against other defined external packages. The system will error if a dependency is ambiguous (matches more than one external) or cannot be satisfied. Dependency types are inferred from the package recipe if not explicitly provided. YAML `dependencies:` Configuration: ---------------------------------- For more complex or unambiguous configurations, a new `dependencies:` block is supported. This provides more explicit control. packages: mpileaks: externals: - spec: "[email protected]" prefix: /user/path dependencies: - id: callpath_id deptypes: [link] - spec: mpich deptypes: [build, link] Dependencies in this block can be specified by `spec:` (which must be unambiguous) or by a unique `id:` that refers to another external (and must also be unambiguous). This format also allows for explicit setting of `deptypes` and `virtuals`. Backward Compatibility (Python): -------------------------------- For now, this change preserves the existing behavior of "stitching" external Python extensions (e..g, `py-foo`) to a single, guessed Python implementation if no explicit dependency is provided. This is maintained to avoid breaking existing configurations, but users can now use the new syntax to explicitly state the Python dependency. Fixes spack#49697 --------- Signed-off-by: Massimiliano Culpo <[email protected]>
This is a hack while we figure out why it's doing this Signed-off-by: Todd Gamblin <[email protected]>
`spack info` has always shown dependencies grouped by type, but not conditional dependency information. This generifies the code used to display variants and uses it for dependencies as well. As with variants, dependencies are shown grouped by conditions, with unconditional dependencies first. - [x] generify when-grouping code to use a `Formatter` class - [x] Rewrite variant code to use generic formatter. - [x] Rewrite dependency code to use generic formatter. - [x] add full type annotations to `cmd/info.py` Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
`spack info` has historically accepted a package name but not a spec.
Now that it shows lots of conditional information, it can be useful
to filter out variants, dependencies, versions, etc. that are irrelevant
to certain package configurations.
- [x] allow `spack info` to take a spec instead of a package name
- [x] filter `spack info` output versions, dependencies, variants, and
licenses based on the input spec
Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
After adding dependencies, it is often much clearer to list info output by name, since there can be a very large number of dependency conditions in the output. `--by-when` is still an option, and the `--by-name` argument is still present even though it is the default. Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
…itions This will advise users to add, e.g., `~cuda` and `~rocm` when output for a package is very long. This makes the feature more discoverable. Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
For complex dependencies like boost, there may be many required variants. Ensure that they don't push deptypes off the edge of the screen by setting a min width and showing deptypes on the next line when the dependency spec is too long. Signed-off-by: Todd Gamblin <[email protected]>
- need to use long_spec for this Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Todd Gamblin <[email protected]>
Signed-off-by: Morten Kristensen <[email protected]>
…ack#51442) Bumps [pylint](https://github.com/pylint-dev/pylint) from 3.3.9 to 4.0.1. - [Release notes](https://github.com/pylint-dev/pylint/releases) - [Commits](pylint-dev/pylint@v3.3.9...v4.0.1) --- updated-dependencies: - dependency-name: pylint dependency-version: 4.0.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
spack#50905 added a few tests that are using the `builtin` repository instead of `builtin.mock`. This is a fix to that small issue. Signed-off-by: Massimiliano Culpo <[email protected]>
…51448) the ConcretizationCache, enabled in spack#50905, could still use a few more simplifications. - [x] Use `len` instead of `sum(1 ...)` - [x] Use `WriteTransaction` and `ReadTransaction` context managers instead of reimplementing. - [x] Clean up fetch a bit for better control flow Signed-off-by: Todd Gamblin <[email protected]>
* spack info: emit a deprecation warning with --variants-by-name - [x] add a `DeprecatedStoreTrueAction` to our common arguments code - [x] use it in `spack info` - [x] update the test for `spack info` to ensure the warning * spack info: minor code cleanup - [x] document max_name_len argument to _print_definition - [x] remove unused type variables - [x] make a comment a bit more future-proof Signed-off-by: Todd Gamblin <[email protected]>
This can be merged once packages spack#342 is merge Signed-off-by: Ryan Krattiger <[email protected]>
* Implement direct pruning for build caches --------- Signed-off-by: Mike VanDenburgh <[email protected]> Co-authored-by: Ryan Krattiger <[email protected]>
* CI: Install fail fast, envvar override for CI Signed-off-by: Ryan Krattiger <[email protected]> * Update completions Signed-off-by: Ryan Krattiger <[email protected]> * Change default to fail-fast true for CI Signed-off-by: Ryan Krattiger <[email protected]> --------- Signed-off-by: Ryan Krattiger <[email protected]>
This PR allows the use of git options, like those supported for `repos.yaml`, in `include.yaml`. This has been tested manually using configuration files from `https://github.com/spack/spack-configs/tree/main/USC/config`. Example (from the updated docs): ``` include: - git: https://github.com/spack/spack-configs branch: main when: os == "centos7" paths: - USC/config/config.yaml - USC/config/packages.yaml ``` - [x] Add unit tests (at least of `git`) options (scopes) --------- Signed-off-by: tldahlgren <[email protected]>
spack#51462) Bumps ruamel-yaml from 0.18.15 to 0.18.16. --- updated-dependencies: - dependency-name: ruamel-yaml dependency-version: 0.18.16 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
If we do not clean up the stage, retries might use a corrupted stage: 1. Cloning a Git repository succeeds 2. Disk space runs out while copying the repository to the stage 3. Since the stage exists, a retry will simply reuse it without noticing that it is missing files due to the previous error Signed-off-by: Michael Kuhn <[email protected]>
Signed-off-by: Harmen Stoppels <[email protected]>
…spack#51717) This reverts commit a6a383e. Signed-off-by: Harmen Stoppels <[email protected]>
some groundwork to make the diff of another pr smaller Signed-off-by: Harmen Stoppels <[email protected]>
This adds support for Python 3.14 in the new installer. The issue is that the classic self-pipe trick does not work for `forkserver`, because there the forkserver process receives the signal that a child has finished, not the parent process. What's more, it avoids the O(n) search for finished subprocesses, whether forkserver or not. Finally it's slightly more non-unix compatible (not that Windows is supported now). Signed-off-by: Harmen Stoppels <[email protected]>
Logging in the new installer was modeled after the old installer, but it has proven to be problematic as it requires the stage dir to exist before a log file is created: 1. many things can go wrong before the stage dir is created 2. for build cache installs pkg.stage is obviously not used, but we do need a log file to troubleshoot failures. In the old installer this was no issue, because install from binary cache was done sequentially from the main process 3. there is output *before* creating the stage dir that needs to be silenced to not mess up the UI This commit generates a unique log file in the stage dir root, redirects output to that as early as possible, and then ultimately symlinks the log file in the package stage dir in the usual location during source builds. Signed-off-by: Harmen Stoppels <[email protected]>
Signed-off-by: Massimiliano Culpo <[email protected]>
Signed-off-by: Harmen Stoppels <[email protected]>
Signed-off-by: Massimiliano Culpo <[email protected]>
…pack#51780) Introduce unique penalties for non-default variant values, following the order of definition in package.py. This leads to deterministic concretization when the default variant value cannot be satisfied or is sub-optimal. Signed-off-by: Massimiliano Culpo <[email protected]>
* ensure the argument type is a tuple * inline `_id` in `__str__` for a minor perf improvement * order branch on argument type by most likely in practice to avoid a chain of isinstance checks * use `type(...) is int` to remove a branch on bool that was only needed with `isinstance(..., int)`. * remove redundant inheritance of `AspObject` Signed-off-by: Harmen Stoppels <[email protected]> * diff.py: fix existing type issue list -> tuple Signed-off-by: Harmen Stoppels <[email protected]> --------- Signed-off-by: Harmen Stoppels <[email protected]>
…pack#51825) Signed-off-by: Gregory Becker <[email protected]>
climbfuji
force-pushed
the
feature/update_spack-stack-dev_from_develop
branch
2 times, most recently
from
ccb0738 to
f98e7c0
Compare
climbfuji
force-pushed
the
feature/update_spack-stack-dev_from_develop
branch
from
f98e7c0 to
459bb8a
Compare
12 tasks
climbfuji
requested review from
AlexanderRichert-NOAA,
eap,
mathomp4 and
rickgrubin-noaa
rickgrubin-noaa
approved these changes
Jan 14, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Note. This PR must be merged as a regular merge, not as a squashed merge!
This PR updates
spack-stack-devfrom the upstream Spackdevelopbranch. Becausespack-stack-devwas previously based on Spack releasev1.0.1with a few modifications cherry-picked from develop, and because the Spack developers no longer maintain mergeability between the releases and the develop branch, I had to resolve several merge conflicts.One of the perks of this update is that we can reenable most GitHub actions tests, which resolves #573.
Because we previously cherry-picked some commits from develop into the spack-stack-dev branch, we now have fewer differences:
spack v1.0.1 vs spack-stack-dev before this PR: https://github.com/spack/spack/compare/v1.0.1...JCSDA:spack:spack-stack-dev?expand=1
spack develop as of 2026/01/09 vs this PR: https://github.com/spack/spack/compare/develop...climbfuji:feature/update_spack-stack-dev_from_develop?expand=1
Testing