issue finding HIP

[berselius]

I am trying to build umpire+rocm in a larger spack env and running into HIP configure issues that I suspect are related to BLT (this is the only project building with it in the env). I have in my packages.py:

 umpire:
    require: '%[when=%c] c=rocmcc %[when=%cxx] cxx=rocmcc %[when=%fortran] fortran=gcc amdgpu_target=gfx942'

where rocmcc is pulling in an externally installed rocm.

In the concretization, I see

umpire@4.1.2~asan~backtrace+c~cuda~dev_benchmarks~device_alloc~deviceconst~examples+fmt_header_only~fortran~ipc_shmem~ipo~mpi~mpi3_shmem~numa~omptarget~openmp+rocm~sanitizer_tests+shared~sqlite_experimental~tools~werror amdgpu_target:=gfx942 build_system=cmake build_type=Release commit=447f4640eff7b8f39d3c59404f3b03629b90c021 generator=make patches:=135bbc7,7d912d3,f3b2133 tests=none platform=linux os=rhel8 target=zen4 %fortran=gcc@13.3.1 %c,cxx=rocmcc@19.0.0
[+]  vn72bsg      ^blt@0.4.1 build_system=generic platform=linux os=rhel8 target=zen4 %fortran=gcc@13.3.1 %c,cxx=rocmcc@19.0.0

which seems like what I would expect. However, when trying to configure umpire, I get

1 error found in build log:
     50    -- Git Executable: /usr/tce/bin/git
     51    -- Git Version: 2.29.1
     52    -- MPI Support is OFF
     53    -- OpenMP Support is OFF
     54    -- CUDA Support is OFF
     55    -- HIP Support is ON
  >> 56    CMake Error at /usr/gapps/user_contrib/capsaicin_modules/spack-environments.20260302/ATS4G_GPU/3.0.1/spack/opt/spack/linux-zen4
           /blt-0.4.1-vn72bsg/cmake/thirdparty/FindHIP.cmake:58 (message):
     57      Specify HIP_ROOT_DIR
     58    Call Stack (most recent call first):
     59      /usr/gapps/user_contrib/capsaicin_modules/spack-environments.20260302/ATS4G_GPU/3.0.1/spack/opt/spack/linux-zen4/blt-0.4.1-vn
           72bsg/cmake/thirdparty/SetupHIP.cmake:13 (find_package)
     60      /usr/gapps/user_contrib/capsaicin_modules/spack-environments.20260302/ATS4G_GPU/3.0.1/spack/opt/spack/linux-zen4/blt-0.4.1-vn
           72bsg/cmake/thirdparty/SetupThirdParty.cmake:56 (include)
     61      /usr/gapps/user_contrib/capsaicin_modules/spack-environments.20260302/ATS4G_GPU/3.0.1/spack/opt/spack/linux-zen4/blt-0.4.1-vn
           72bsg/SetupBLT.cmake:112 (include)
     62      CMakeLists.txt:120 (include)

What might I be missing here, or is there some other secret sauce I'm missing to build this?

[berselius]

ope, I think the issue was just that spack concretized to some ancient version of umpire for some reason

[white238]

Yea I was gonna say. We heavily reworked the BLT support for HIP quite a bit after that Umpire version. If you can it would be best to update your version of Umpire/BLT.

Also I don't see a +hip (this is wrong it's +rocmis what I meant) in the concretization of Umpire. Please make sure that is there after you update, but it could be related to the version.

If you need more help, please reach out here or in my email.

[berselius]

Ah, I expected +rocm to give what I wanted. It doesn't look like there is a +hip variant?

[berselius]

hmm, it looks like spack/cmake is trying to build umpire in a cuda flavored way, or at least, trying to grab something from cuda_wrappers

    56        Building HIP object CMakeFiles/cmTC_80b89.dir/testHIPCompiler.hip.o
     57        /opt/rocm-6.4.2/bin/amdclang++  -D__HIP_ROCclr__=1 -isystem /opt/rocm-6.4.2/include --gcc-toolchain=/opt/rocm-6.4.2 --cuda-h
           ost-only  -std=gnu++14 --offload-arch=gfx942 -o CMakeFiles/cmTC_80b89.dir/testHIPCompiler.hip.o -x hip -c /tmp/haack4/spack-stag
           e/spack-stage-umpire-2025.03.0-oirwrioczbdpuqeiseqyagnxhqe73mro/spack-build-oirwrio/CMakeFiles/CMakeScratch/TryCompile-Fd1PUN/te
           stHIPCompiler.hip
     58        In file included from <built-in>:1:
     59        In file included from /opt/rocm-6.4.2/lib/llvm/lib/clang/19/include/__clang_hip_runtime_wrapper.h:111:
  >> 60    /opt/rocm-6.4.2/lib/llvm/lib/clang/19/include/cuda_wrappers/cmath:27:15: fatal error: 'cmath' file not found
     61           27 | #include_next <cmath>
     62              |               ^~~~~~~

EDIT or perhaps just a bad toolchain file is being set

[berselius]

After more hacking, I suspect something in the BLT process is setting the incorrect toolchain in the generated initconfig .cmake file rzadams-toss_4_x86_64_ib_cray-llvm-amdgpu@19.0.0-dmptnuzs.cmake

set(CMAKE_HIP_FLAGS "--gcc-toolchain=/opt/rocm-6.4.2" CACHE STRING "")

But I don't know enough about the internals of BLT / the spack initiconfig step to figure out how to change it. Is there some variable I should be setting elsewhere in my spack env config or similar?

[white238]

Ah, I expected +rocm to give what I wanted. It doesn't look like there is a +hip variant?

Sorry you are right, I meant +rocm and I do see that in the original spec. I should let my caffeine kick in before responding.

[kennyweiss]

Perhaps something in this toss4_cray environment from Axom can help?
https://github.com/llnl/axom/blob/develop/scripts/spack/configs/toss_4_x86_64_ib_cray/spack.yaml

[berselius]

hmm, nothing in particular seems to be jumping out at me. The llvm-amdgpu setup looks pretty similar to what I have.

[white238]

I am not confident the versions of BLT and Umpire you listed above are going to work. In fact this reminds me I need to deprecate a bunch of versions across multiple projects in Spack. Are you able to update them?

[berselius]

I can hack something in if needed for now (right now I have 2025.09.0 and I want to say 0.7.1)

[white238]

I dont know where CMAKE_HIP_FLAGS would be coming from.. i only see this line:

https://github.com/spack/spack-packages/blob/e2beb58c7acda74b234c8fc419ed865600d24c36/repos/spack_repo/builtin/packages/umpire/package.py#L435