Skip to content

Use shallow fetch for faster bootstrapping#1381

Open
linsword13 wants to merge 1 commit into
llnl:developfrom
linsword13:optimize-checkout-fetch
Open

Use shallow fetch for faster bootstrapping#1381
linsword13 wants to merge 1 commit into
llnl:developfrom
linsword13:optimize-checkout-fetch

Conversation

@linsword13

Copy link
Copy Markdown
Contributor

Description

Use shallow fetch for faster bootstrapping

The checkout-versions.yaml is updated to use the full commit-sha, as required by remote shallow fetch.

Here's a timing comparison:

  • With the change
$ rm -rf scratch/
$ time benchpark list experiments
Cloning Ramble to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/ramble
Cloning packages to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack-packages
Cloning Spack to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack
Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    ad+[mpi]
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    branson+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    commbench+[cuda|rocm]
    genesis+[openmp|mpi]
    gpcnet+[mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
    hpl+[openmp|mpi]+[strong|weak]
    ior+[mpi]+[strong|weak]
    kripke+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    laghos+[cuda|rocm|mpi]+[strong|weak|throughput]
    lammps+[openmp|cuda|rocm|mpi]+[strong]
    md-test+[mpi]+[strong]
    osu-micro-benchmarks+[cuda|rocm|mpi]
    phloem+[mpi]
    py-scaffold+[cuda|rocm]+[strong|weak]
    quicksilver+[openmp|mpi]+[strong|weak]
    qws+[openmp|mpi]+[strong|weak|throughput]
    raja-perf+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    remhos+[cuda|rocm|mpi]+[strong|weak|throughput]
    salmon-tddft+[openmp|mpi]
    smb+[mpi]
    sparta-snl+[openmp|cuda|rocm|mpi]
    stream+[mpi]
    test

real	0m7.585s
user	0m1.769s
sys	0m2.374s
  • Without the change
$ git stash
Saved working directory and index state WIP on optimize-checkout-fetch: b1055bd `lanl-venado` change h100 -> gh200 (#1379)
$ rm -rf scratch/
$ time benchpark list experiments
Cloning Ramble to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/ramble
Cloning packages to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack-packages
Cloning Spack to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack
Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    ad+[mpi]
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    branson+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    commbench+[cuda|rocm]
    genesis+[openmp|mpi]
    gpcnet+[mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
    hpl+[openmp|mpi]+[strong|weak]
    ior+[mpi]+[strong|weak]
    kripke+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    laghos+[cuda|rocm|mpi]+[strong|weak|throughput]
    lammps+[openmp|cuda|rocm|mpi]+[strong]
    md-test+[mpi]+[strong]
    osu-micro-benchmarks+[cuda|rocm|mpi]
    phloem+[mpi]
    py-scaffold+[cuda|rocm]+[strong|weak]
    quicksilver+[openmp|mpi]+[strong|weak]
    qws+[openmp|mpi]+[strong|weak|throughput]
    raja-perf+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    remhos+[cuda|rocm|mpi]+[strong|weak|throughput]
    salmon-tddft+[openmp|mpi]
    smb+[mpi]
    sparta-snl+[openmp|cuda|rocm|mpi]
    stream+[mpi]
    test

real	1m10.219s
user	3m2.025s
sys	0m12.604s

@github-actions github-actions Bot added the feature New feature or request label Jun 9, 2026
@linsword13

Copy link
Copy Markdown
Contributor Author

All the failing tests seem to be tied to coverage upload. I am not sure if that's related to the current change though.

error - 2026-06-09 05:29:03,805 -- Upload queued for processing failed: {"message":"Token required - not valid tokenless upload"}
Sentry is attempting to send 1 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit
==> Failed to run upload-coverage
    Exiting...

The `checkout-versions.yaml` is updated to use the full commit-sha, as required by remote shallow fetch.

Here's a timing comparison:

* With the change

```
$ rm -rf scratch/
$ time benchpark list experiments
Cloning Ramble to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/ramble
Cloning packages to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack-packages
Cloning Spack to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack
Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    ad+[mpi]
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    branson+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    commbench+[cuda|rocm]
    genesis+[openmp|mpi]
    gpcnet+[mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
    hpl+[openmp|mpi]+[strong|weak]
    ior+[mpi]+[strong|weak]
    kripke+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    laghos+[cuda|rocm|mpi]+[strong|weak|throughput]
    lammps+[openmp|cuda|rocm|mpi]+[strong]
    md-test+[mpi]+[strong]
    osu-micro-benchmarks+[cuda|rocm|mpi]
    phloem+[mpi]
    py-scaffold+[cuda|rocm]+[strong|weak]
    quicksilver+[openmp|mpi]+[strong|weak]
    qws+[openmp|mpi]+[strong|weak|throughput]
    raja-perf+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    remhos+[cuda|rocm|mpi]+[strong|weak|throughput]
    salmon-tddft+[openmp|mpi]
    smb+[mpi]
    sparta-snl+[openmp|cuda|rocm|mpi]
    stream+[mpi]
    test

real	0m7.585s
user	0m1.769s
sys	0m2.374s
```

* Without the change

```
$ git stash
Saved working directory and index state WIP on optimize-checkout-fetch: b1055bd `lanl-venado` change h100 -> gh200 (llnl#1379)
$ rm -rf scratch/
$ time benchpark list experiments
Cloning Ramble to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/ramble
Cloning packages to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack-packages
Cloning Spack to /Users/linsword/hpc/contrib/benchpark/scratch/benchpark_home_fallback/spack
Experiments - BENCHMARK+PROGRAMMING_MODEL+SCALING
    ad+[mpi]
    amg2023+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    babelstream+[openmp|cuda|rocm]
    branson+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    commbench+[cuda|rocm]
    genesis+[openmp|mpi]
    gpcnet+[mpi]
    gromacs+[openmp|cuda|rocm|mpi]
    hpcg+[openmp|mpi]+[strong|weak]
    hpl+[openmp|mpi]+[strong|weak]
    ior+[mpi]+[strong|weak]
    kripke+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    laghos+[cuda|rocm|mpi]+[strong|weak|throughput]
    lammps+[openmp|cuda|rocm|mpi]+[strong]
    md-test+[mpi]+[strong]
    osu-micro-benchmarks+[cuda|rocm|mpi]
    phloem+[mpi]
    py-scaffold+[cuda|rocm]+[strong|weak]
    quicksilver+[openmp|mpi]+[strong|weak]
    qws+[openmp|mpi]+[strong|weak|throughput]
    raja-perf+[openmp|cuda|rocm|mpi]+[strong|weak|throughput]
    remhos+[cuda|rocm|mpi]+[strong|weak|throughput]
    salmon-tddft+[openmp|mpi]
    smb+[mpi]
    sparta-snl+[openmp|cuda|rocm|mpi]
    stream+[mpi]
    test

real	1m10.219s
user	3m2.025s
sys	0m12.604s
```
@linsword13 linsword13 force-pushed the optimize-checkout-fetch branch from a93c372 to 1e7be11 Compare June 17, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant