Increase required C++ version to C++23 and CUDA version to 12.4

[jngrad]

New features in C++23

Relevant to ESPResSo:

• std::ranges::fold_left() for range-based reductions
• std::string::contains()
• std::optional monadic operations
• std::ranges::views::zip (but zip transform not available in Clang < 22)
• std::ranges::views::join_with() array concatenation (GCC >= 13, Clang >= 21)
• std::unreachable()
• std::to_underlying() to convert enum class values to integers without cast
• deducing this aka "self pointer" (not implemented in GCC 13)
• size_t literal suffix
• [P2162R2] which fully specifies visitation of recursive variants

Relevant to ESPResSo but not available yet:

• std::mdspan multi-dimensional array view (not implemented by GCC)
• std::basic_stacktrace() Python-like stack trace (not implemented by any vendor)
• std::ranges::to() for STL containers built from ranges (not implemented by any vendor)

There are other features available in C++23, but I did not find a clear-cut application in ESPResSo.

Applicability

All aforementioned features have use cases in ESPResSo, but are only quality-of-life improvements. std::ranges::fold_left would help remove the last remaining Boost algorithms. Literal suffixes would reduce the verbosity of variable declarations in counter-based loops (it's easy to forget adding the std:: prefix to size_t, in which case compiler diagnostics can be emitted). Deducing this would remove the need for dynamic_cast in a few places.

We would need to drop support for GCC 12, which is still available on HPC clusters. This is discussed in more details further below. Dropping GCC 12 would also allow us to replace all calls to std::to_string by corresponding calls to std::format, a change that couldn't be carried out during the move to C++20 due to poor compiler support 2 years ago. The standard function std::to_string is known to be ill-suited to represent floating-point numbers outside the range [1e6, 1e-6] due to the decimal format it uses, among other issues outlined in draft proposal [D2587R1]. In fact, std::to_string is redefined around std::format in C++26, but without addressing the underlying issue of precision loss in small numbers. std::to_string is used 70 times in ESPResSo due to its convenience over alternatives based on std::stringstream or snprintf, approximately 20 of which involve a floating-point value. Likewise, several calls to std::stringstream are used to format error messages and could be replaced by a corresponding std::format call.

Discussion

Issue: cannot build an application with Boost and CUDA using the CMake build system [solved]

Solution: use CXX=g++-15 CUDAHOSTCXX=g++-13 cmake .. instead of -D CMAKE_CUDA_FLAGS="--compiler-bindir=/usr/bin/g++-13" and add target_link_directories(target BEFORE PUBLIC /usr/lib/gcc/x86_64-linux-gnu/14) to targets

Problem statement: Boost.mpi and Boost.serialization require symbols only available in libstdc++ 14 and up, and nvcc cannot use GCC 14 and GCC 15. While manually linking with GCC 14 or GCC 15 is possible as a workaround, we need solution that is natively supported by the build system. Passing --compiler-bindir=/usr/bin/g++-13 to nvcc is mandatory to select a supported GCC compiler for host code. CMake picks this up and sets internal variable CMAKE_CUDA_HOST_LINK_LAUNCHER to /usr/bin/g++-13, but we need /usr/bin/g++-14 or /usr/bin/g++-15 as the linker. Manually editing that internal variable in ./CMakeFiles/3.31.6/CMakeCUDACompiler.cmake solves the issue, but goes against best practices and will break in future CMake versions.

The proper way to do this is through the combination of:

• ENV{CUDAHOSTCXX} (or -D CMAKE_CUDA_HOST_COMPILER) to select GCC 13 as the host compiler (CMake will forward it as a --compiler-bindir/--ccbin nvcc flag)
• ENV{CXX} to select GCC 15 as the linker (and C++ compiler)

This is documented in the user guide section 2.2.1. Installing requirements on Ubuntu (scroll down until the box labeled "GCC Ubuntu 26.04" is visible, right above section 2.2.1.2).

Depending on how CMake was packaged, there is still a possibility for CMAKE_CUDA_HOST_LINK_LAUNCHER to be set to the value of CMAKE_CUDA_HOST_COMPILER. Gentoo introduced a new CMake option CUDAHOSTLD to give control over the value of CMAKE_CUDA_HOST_LINK_LAUNCHER (patch).

minimal working examples (click to unroll)

MWE with ESPResSo:

$ touch myconfig.hpp
$ CC=gcc-15 CXX=g++-15 CUDAARCHS="75;86" cmake .. -D ESPRESSO_BUILD_WITH_CUDA=ON -D CMAKE_CUDA_FLAGS="--compiler-bindir=/usr/bin/g++-13" -D ESPRESSO_BUILD_WITH_WALBERLA=OFF -D ESPRESSO_BUILD_WITH_FFTW=OFF
$ make
$ make unit_tests_executables
/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.88.0: undefined reference to `__cxa_call_terminate@CXXABI_1.3.15'
collect2: error: ld returned 1 exit status
$ strings /usr/lib/x86_64-linux-gnu/libboost_mpi.so.1.88.0 | grep -P "CXXABI_|__cxa_call_terminate"
__cxa_call_terminate
CXXABI_1.3
CXXABI_1.3.8
CXXABI_1.3.9
CXXABI_1.3.15
$ strings /lib/gcc/x86_64-linux-gnu/13/libstdc++.so | grep -P "CXXABI_|__cxa_call_terminate"
CXXABI_1.3
CXXABI_1.3.1
...
CXXABI_1.3.14
$ strings /lib/gcc/x86_64-linux-gnu/14/libstdc++.so | grep -P "CXXABI_|__cxa_call_terminate"
__cxa_call_terminate
CXXABI_1.3
CXXABI_1.3.1
...
CXXABI_1.3.14
CXXABI_1.3.15

This affects C++ unit tests but not ESPResSo shared libraries when no external dependency is pulled in.

MWE without ESPResSo:

==> CMakeLists.txt <==
cmake_minimum_required(VERSION 3.27.6)
cmake_policy(VERSION 3.27.6)
project(MyProject VERSION "5.0.0" LANGUAGES CXX CUDA)
find_package(CUDAToolkit 12.0 REQUIRED)
add_library(core SHARED a.cpp a.cu)
# override search path for libstdc++.so to reproduce the linker error
target_link_directories(core BEFORE PUBLIC /usr/lib/gcc/x86_64-linux-gnu/13)
# override search path for libstdc++.so to select the correct shared library: doesn't work (removed as duplicate)
target_link_directories(core BEFORE PUBLIC /usr/lib/gcc/x86_64-linux-gnu/14)
add_executable(main main.cpp)
target_link_libraries(main PRIVATE core)

==> a.cpp <==
int foo();
void err() {
  __cxa_call_terminate(nullptr);
}
int bar() {
  return foo();
}

==> a.cu <==
int foo() {return 1;}

==> main.cpp <==
int bar();
int main(){return bar();}

mkdir build
cd build
CXX=g++-14 CUDAHOSTCXX=g++-13 cmake ..
make main VERBOSE=1

Output:

-- The CXX compiler identification is GNU 14.3.0
-- The CUDA compiler identification is NVIDIA 12.4.131 with host compiler GNU 13.4.0
-- Check for working CXX compiler: /usr/bin/g++-14 - skipped
-- Check for working CUDA compiler: /usr/bin/nvcc - skipped
-- Found CUDAToolkit: /usr/include (found suitable version "12.4.131", minimum required is "12.0")
-- Configuring done (0.9s)
-- Generating done (0.0s)
[ 20%] Building CXX object CMakeFiles/core.dir/a.cpp.o
/usr/bin/g++-14 -Dcore_EXPORTS  -fPIC -MD -MT CMakeFiles/core.dir/a.cpp.o -MF CMakeFiles/core.dir/a.cpp.o.d -o CMakeFiles/core.dir/a.cpp.o -c /home/espresso/espresso/gcc14/a.cpp
[ 40%] Building CUDA object CMakeFiles/core.dir/a.cu.o
/usr/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/usr/bin/g++-13 -Dcore_EXPORTS  "--generate-code=arch=compute_52,code=[compute_52,sm_52]" -Xcompiler=-fPIC -MD -MT CMakeFiles/core.dir/a.cu.o -MF CMakeFiles/core.dir/a.cu.o.d -x cu -c /home/espresso/espresso/gcc14/a.cu -o CMakeFiles/core.dir/a.cu.o
[ 60%] Linking CXX shared library libcore.so
/usr/bin/g++-14 -fPIC -Wl,--dependency-file=CMakeFiles/core.dir/link.d -shared -Wl,-soname,libcore.so -o libcore.so CMakeFiles/core.dir/a.cpp.o CMakeFiles/core.dir/a.cu.o   -L/usr/lib/gcc/x86_64-linux-gnu/13  -L/usr/lib/x86_64-linux-gnu/stubs  -lcudadevrt -lcudart_static -lrt -lpthread -ldl
[ 80%] Building CXX object CMakeFiles/main.dir/main.cpp.o
/usr/bin/g++-14    -MD -MT CMakeFiles/main.dir/main.cpp.o -MF CMakeFiles/main.dir/main.cpp.o.d -o CMakeFiles/main.dir/main.cpp.o -c /home/espresso/espresso/gcc14/main.cpp
[100%] Linking CXX executable main
/usr/bin/g++-14 -Wl,--dependency-file=CMakeFiles/main.dir/link.d CMakeFiles/main.dir/main.cpp.o -o main   -L/usr/lib/gcc/x86_64-linux-gnu/13  -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13:/home/espresso/espresso/gcc14/build libcore.so
/usr/bin/x86_64-linux-gnu-ld.bfd: libcore.so: undefined reference to `__cxa_call_terminate'

C++ standard version

ESPResSo is C++23-ready since d42029c and supports CUDA 12.4 to 12.8 since 1f838ac.

C++23 is not supported by nvcc <= 13.2 (support table). We would need to build C++ code in C++23 mode and host/device code in C++20 mode. Linking object files together from different STL versions (libstdc++/libc++) is specifically allowed by library vendors, since it's part of ABI compatibility. This is why libstdc++ ships two completely different implementations of std::string, one for C++98 and one for C++11. The best resource I could find on the subject is this SO post by a maintainer of the libstdc++ library. The only caveat is that we cannot link two objects when one is built with a compiler that has partial support for a feature and the other is built with complete support. There is no document tracking the state of support completeness, other that the contributed list at cppreference. We already do mix C++17 and C++20 objects in ESPResSo (Kokkos builds C++ and CUDA code in C++17 mode), and our CUDA code is quite conservative: no experimental features and STL container usage is limited to std::span (C++20), std::optional (C++17), std::array (C++11), std::vector (C++98).

I do not see an obstacle to using C++23 in C++ source files as long as we require GCC 14 or 15 in C++ code and GCC 13 in CUDA host code. Building ESPResSo that way and running the testsuite did not reveal any linker error or runtime error.

NumPy version support

ESPResSo 5.0 supports both NumPy 1.26.4 and NumPy 2.0. Since 1.26.4 is the final release in the 1.x line and is 2 years old (released Feb 6, 2024), now is a good time to push for NumPy 2.x. My take on this is to target NumPy >= 2.2, which is satisfied by Debian Trixie, Ubuntu 26.04, Fedora, and EESSI 2025.06.

Toolchain updates

CUDA 12.4 officially supports GCC 6.x to 13.2 and Clang 7.x to 17.0 (Table 2: Supported Compilers). However, GCC version 13.4 works too, and Clang versions 19/20/21 work with a compiler warning about the unrecognized CUDA version (silenced with -Wno-unknown-cuda-version). With GCC 14.3 and 15.2, a compiler error is triggered by the bfloat16_t type.

My take on this is to support GCC 13 for CUDA code (in C++20 mode) and GCC 14 for C++ code (in C++23 mode), and Clang >= 19 for C++/CUDA code.

• waLBerla dropped GCC 13/14 and Clang 19 from CI (walberla/walberla!800)
• HPC LUMI provides CrayClang 19 since January 2026
• HPC MareNostrum has CUDA 12.5 with nvhpc 25.7 and GCC 13 on the ACC partition, Intel oneAPI 2025.1 on the GPP partition
• EESSI: foss/2025a provides GCC 14.2 and Clang 20, we won't be using foss/2023b for ESPResSo 5.1 since dependencies are too old

Course of action

• Update ESPRESSO_MINIMAL_CXX_STANDARD to 23
• Keep ESPRESSO_MINIMAL_CUDA_STANDARD to 20
• Drop support for CUDA <= 12.3
• Drop support for NumPy <= 2.1
• Don't increase minimal Boost version (for HPC users and Ubuntu 24.04 LTS users)

Impact on users and risk mitigation:

• Ubuntu 24.04 LTS users
  • GCC 13, GCC 14 and Clang 19 are already available
  • CUDA 12.4 isn't available but can be installed with root privileges
  • the ESPResSo 5.0.x line will continue supporting CUDA 12.0
• NumPy >= 2.2 is trivial to install, now that virtual environments are mandatory on all major Linux operating systems
• most HPC clusters equipped with CUDA hardware already provide CUDA 12.4
• several HPC clusters provide access to recent GCC/Clang toolchains via EESSI (full list)
• HPC toolchains survey:
  • MareNostrum GPP: IntelLLVM 2023.2/2024.2/2025.2
  • MareNostrum ACC: NVHPC 25.7 + GCC 13.2 + CUDA 12.9
  • LUMI: CrayClang 19 (HPE Cray 25.03) + ROCm
  • Vega: CUDA 12.7
  • Deucalion: GCC 13/14, CUDA 12.4/12.6/12.9
  • Meluxina: GCC 14.3 + Clang 20.1 + CUDA 12.8
  • JUPITER: NVHPC 25.9 + GCC 14.3 + CUDA 13
  • Karolina: unknown compiler toolchain versions, but has EESSI with GPU support
  • Leonardo: GCC 12.2 + CUDA 12.2 (won't work, and doesn't have EESSI)
  • Helix: GCC 14.1 + CUDA 12.9
  • BinAC2: GCC 14.2 + CUDA 12.8
  • bwUniCluster3: gompi/2025a + CUDA 12.8
  • EESSI foss/2025a and gompi/2025a: GCC 14.3 + Clang 20.1 + CUDA 12.6/12.8

[RudolfWeeber]

I think, using different c++ standard versions in Cuda might get us intro trouble, as a lot of Kokkos-based code will in the end be compiled, both for the CPU and the GPU. Keeping the include-chian of these file sc++-23 free might become a non-obvious task....

[jngrad]

The Kokkos CUDA support introduced by #5400 forces us to stay with C++20 for now, since every translation unit that includes Kokkos needs to be built by nvcc, unless Clang is used as device code compiler. This creates an unsustainable situation for us: only the Clang build in our CI pipelines can build ESPResSo with Kokkos CUDA support, since the GCC builds rely on nvcc 12.0, which is not supported by Kokkos anymore. Downgrading Kokkos to 4.7.4 doesn't work, since nvcc 12.0 has limited support for template programming and cannot compile our heavily-templated P3M code. The Boost library has also become problematic, since several Boost header files have CUDA-enabled code paths that don't compile in a C++ file when processed by the nvcc compiler.

CUDA 13.3 introduces support for C++23, but isn't widely available yet.