Trilinos/Muelu Best Practices for Intel -spr builds?

[spdomino]

Question

Greetings,

I am running a research-based simulation study on the NSF TACC Stampeded3 partition, see: https://docs.tacc.utexas.edu/hpc/stampede3/

Are there any best practices for building on an Intel-based Saphire Rapids? This machine mirrors an institutional resource at the Labs (112 cores on two sockets (2 x 56 cores)) where we successfuly ran Nalu (200 node, 600 Mil element mesh) on a production simulation of an elevated cylinder subjected to crossflow in both the sub and supercritical drag regime. I avoided any -spr internal flags as I wanted to be able to run the same executable over the Stampede resource (SPR, SPX, SKX). What I found is that we have Muelu dying in a platform memory allocation - I suppose not unlike what I reported a few months ago, but again at the Muelu construction. Sadly, all I see are system errors "bad allocation".

Second, and just as complicated, I backed down to the TACC ICX partition (dual socket, 40 cores/socket), simulation time seem roughly the same speed as a legacy Stanford platform (circa 2015) of Xenon nodes, 32 cores on two sockets (2x16 cores using hyper threading; matching the overall core count of O(1000)).

Something looks painfully apparent in our continuity solve with XML as follows: https://github.com/COMERI/Mare-Nalu/blob/main/reg_tests/xml/milestone.xml When I say painfully apparent, I mean like Haswell vs KNL like shock:)

I am retreating to my Trilinos build in late March (the same build the Stanford resource is using) to see if there was a performance regression (I know I have reported on a few diffs over the months; and I doubt the latest Intrepid ticket is related to this - but I need to try something).

Any suggestions on either nuance (SPR first, comments on performance regression second) would be appreciated. Are there any SPR tests in-house to the development process?

# The base directory where mpi is located.  
# From here you should be able to find include/mpi.h bin/mpicxx, bin/mpiexec, etc.
mpi_base_dir=/opt/intel/oneapi/mpi/2021.11/bin/mpicc/bin
nalu_install_dir=/work/11662/sdomino/stampede3/scratch_build/install/intel_24.0_impi_21.11

# Note: Don't forget to set your LD_LIBRARY_PATH to $mpi_base_dir/lib
#       You may also need to add to LD_LIBRARY_PATH the lib directory for the compiler
#       used to create the mpi executables.

# TPLS needed by trilinos, possibly provided by HomeBrew on a Mac
netcdf_install_dir=$nalu_install_dir/netcdf/4.9.3
hdf_install_dir=$nalu_install_dir/hdf5/1.13.0
pnetcdf_install_dir=$nalu_install_dir/pnetcdf/1.13.0
gtest_install_dir=/usr/local/lib/

# libraries for BLAS and LAPACK
blas_install_lib="${MKLROOT}/lib/intel64/libmkl_intel_lp64.so\;${MKLROOT}/lib/intel64/libmkl_sequential.so\;${MKLROOT}/lib/intel64/libmkl_
core.so"
lapack_install_lib="${MKLROOT}/lib/intel64/libmkl_intel_lp64.so\;${MKLROOT}/lib/intel64/libmkl_sequential.so\;${MKLROOT}/lib/intel64/libmk
l_core.so"

# Where trilinos will be installed
trilinos_install_dir=$nalu_install_dir/Trilinos_stable_release

echo "nalu_install_dir = \"$nalu_install_dir\""

EXTRA_ARGS=$@

# Cleanup old cache before we configure
# Note:  This does not remove files produced by make.  Use "make clean" for this.
find . -name "CMakeFiles" -exec rm -rf {} \;
rm -f CMakeCache.txt

cmake \
-DCMAKE_INSTALL_PREFIX=$trilinos_install_dir \
-DCMAKE_CXX_STANDARD:STRING=20 \
-DTrilinos_ENABLE_CXX11=ON \
-DCMAKE_BUILD_TYPE=RELEASE \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-DTpetra_INST_DOUBLE:BOOL=ON \
-DTpetra_INST_INT_LONG:BOOL=ON \
-DTpetra_INST_INT_LONG_LONG:BOOL=OFF \
-DTpetra_INST_COMPLEX_DOUBLE=OFF \
-DTrilinos_ENABLE_TESTS:BOOL=OFF \
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-DTrilinos_ALLOW_NO_PACKAGES:BOOL=OFF \
-DTPL_ENABLE_MPI=ON \
  -DMPI_BASE_DIR:PATH=$mpi_base_dir \
-DTPL_ENABLE_SuperLU=OFF \
-DTPL_ENABLE_Boost:BOOL=OFF \
-DTrilinos_ENABLE_Epetra:BOOL=OFF \
-DTrilinos_ENABLE_Tpetra:BOOL=ON \
-DTrilinos_ENABLE_ML:BOOL=OFF \
-DTrilinos_ENABLE_MueLu:BOOL=ON \
-DTrilinos_ENABLE_Stratimikos:BOOL=OFF \
-DTrilinos_ENABLE_Thyra:BOOL=OFF \
-DTrilinos_ENABLE_EpetraExt:BOOL=OFF \
-DTrilinos_ENABLE_AztecOO:BOOL=OFF \
-DTrilinos_ENABLE_Belos:BOOL=ON \
-DTrilinos_ENABLE_Ifpack2:BOOL=ON \
-DTrilinos_ENABLE_Amesos2:BOOL=ON \
-DTrilinos_ENABLE_Zoltan2:BOOL=ON \
-DTrilinos_ENABLE_Ifpack:BOOL=OFF \
-DTrilinos_ENABLE_Amesos:BOOL=OFF \
-DTrilinos_ENABLE_Zoltan:BOOL=ON \
-DTrilinos_ENABLE_STKMesh:BOOL=ON \
-DTrilinos_ENABLE_STKSimd:BOOL=ON \
-DTrilinos_ENABLE_STKIO:BOOL=ON \
-DTrilinos_ENABLE_STKTransfer:BOOL=ON \
-DTrilinos_ENABLE_STKSearch:BOOL=ON \
-DTrilinos_ENABLE_STKUtil:BOOL=ON \
-DTrilinos_ENABLE_STKTopology:BOOL=ON \
-DTrilinos_ENABLE_STKBalance:BOOL=OFF \
-DTrilinos_ENABLE_STKUnit_tests:BOOL=OFF \
-DTrilinos_ENABLE_STKUnit_test_utils:BOOL=OFF \
-DTrilinos_ENABLE_gtest:BOOL=OFF \
-DTrilinos_ENABLE_SEACASExodus:BOOL=ON \
-DTrilinos_ENABLE_SEACASEpu:BOOL=ON \
-DTrilinos_ENABLE_SEACASExodiff:BOOL=ON \
-DTrilinos_ENABLE_SEACASNemspread:BOOL=ON \
-DTrilinos_ENABLE_SEACASNemslice:BOOL=ON \
-DTrilinos_ENABLE_SEACASIoss:BOOL=ON \
-DKokkos_ENABLE_ATOMICS_BYPASS=ON \
-DKokkos_ENABLE_DEPRECATED_CODE_4=OFF \
-DTPL_ENABLE_Netcdf:BOOL=ON \
-DNetCDF_ROOT:PATH=${netcdf_install_dir} \
-DHDF5_ROOT:PATH=${hdf_install_dir} \
-DHDF5_NO_SYSTEM_PATHS=ON \
-DPNetCDF_ROOT:PATH=${pnetcdf_install_dir} \
-DGTest_DIR:PATH=${gtest_install_dir} \
-DTPL_BLAS_LIBRARIES=${blas_install_lib} \
-DTPL_LAPACK_LIBRARIES=${lapack_install_lib} \
$EXTRA_ARGS \
../

[github-actions[bot]]

Automatic mention of the @trilinos/muelu team

[jhux2]

@spdomino We maintain some performance runs on Aurora (ALCF). Here is the module environment & configuration for those runs:

modules

  1) libfabric/1.22.0
  2) cray-pals/1.8.0
  3) cray-libpals/1.8.0
  4) gcc/13.4.0
  5) oneapi/release/2025.3.1
  6) mpich/opt/develop-git.6037a7a
  7) gcc-runtime/13.4.0-2tg3zy7            (H)
  8) bzip2/1.0.8
  9) intel-oneapi-runtime/2025.3.1-h4uj4w3 (H)
 10) zstd/1.5.7-dyoofjb                    (H)
 11) netcdf-c/4.9.3
 12) parallel-netcdf/1.14.1
 13) hdf5/1.14.6
 14) icu4c/74.2-on2hwtu                    (H)
 15) expat/2.7.3-6auksxt                   (H)
 16) gdbm/1.25
 17) sqlite/3.50.4-s65ksrg                 (H)
 18) util-linux-uuid/2.41-ytoy2bg          (H)
 19) python/3.12.12
 20) boost/1.88.0
 21) cmake/3.31.11
 22) ninja/1.13.0

cmake options

export KOKKOS_DEVICES=SYCL
export KOKKOS_ARCH=PVC,SPR
export BUILD_TYPE="RELEASE"

CMAKE_OPTIONS=(
    -GNinja
    -D Trilinos_TEST_CATEGORIES=PERFORMANCE
    -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF
    -D CMAKE_CXX_COMPILER:STRING="mpicxx"
    -D CMAKE_BUILD_TYPE:STRING="$BUILD_TYPE"
    -D BUILD_SHARED_LIBS:BOOL=ON
    -D CMAKE_POSITION_INDEPENDENT_CODE=ON
    -D CMAKE_CXX_FLAGS:STRING=""
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON
    -D Trilinos_ENABLE_TESTS=ON
    -D Trilinos_ENABLE_EXAMPLES=ON

    -D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-ppn;12;--depth=1;--cpu-bind;depth;gpu_tile_compact.sh"

    -D Kokkos_ARCH_SPR:BOOL=ON
    -D Kokkos_ARCH_INTEL_PVC:BOOL=ON
    -D Kokkos_ENABLE_SERIAL:BOOL=ON
    -D Kokkos_ENABLE_SYCL:BOOL=ON
    -D Kokkos_ENABLE_DEBUG:BOOL=OFF
    -D Kokkos_ENABLE_DEBUG_BOUNDS_CHECK:BOOL=OFF
    -D Kokkos_ENABLE_TESTS:BOOL=OFF
    -D Kokkos_ENABLE_EXAMPLES:BOOL=OFF
    -D Kokkos_ENABLE_ONEDPL=OFF

    -D Tpetra_INST_SYCL:BOOL=ON
    -D Tpetra_INST_SERIAL=ON

    -D Trilinos_ENABLE_Amesos2:BOOL=ON
    -D Trilinos_ENABLE_Belos=ON
    -D Trilinos_ENABLE_Ifpack2=ON
    -D Trilinos_ENABLE_Intrepid2=ON
    -D Trilinos_ENABLE_MueLu=ON
    -D Trilinos_ENABLE_PanzerMiniEM=ON
    -D Trilinos_ENABLE_PanzerAdaptersSTK=ON
    -D Trilinos_ENABLE_Percept=OFF
    -D Trilinos_ENABLE_STKIO=ON
    -D TPL_ENABLE_gtest=OFF
    -D STK_ENABLE_TESTS=OFF
    -D Trilinos_ENABLE_Tpetra=ON
    -D Trilinos_ENABLE_Thyra=ON
     -D  Trilinos_ENABLE_ThyraTpetraAdapters=ON
    -D Trilinos_ENABLE_Xpetra=ON
    -D Trilinos_ENABLE_Zoltan2=ON

     -D Trilinos_ENABLE_ShyLU_DDCore=ON
     -D Trilinos_ENABLE_ShyLU_DDFROSch=ON

    -D Trilinos_ENABLE_NOX=ON
     -D NOX_ENABLE_ThyraCore=ON
     -D NOX_ENABLE_Stratimikos=ON
    -D Trilinos_ENABLE_Stratimikos=ON
     -D Stratimikos_ENABLE_Belos=ON
    -D Trilinos_ENABLE_PyTrilinos=OFF

    -D TPL_ENABLE_Matio=OFF
    -D TPL_ENABLE_X11=OFF
    -D TPL_ENABLE_CGNS=OFF
    -D TPL_ENABLE_MPI=ON

    -D Trilinos_ENABLE_Boost=ON
     -D Boost_INCLUDE_DIRS=${BOOST_ROOT}/include
    -D Trilinos_ENABLE_BoostLib=ON
     -D BoostLib_INCLUDE_DIRS=${BOOST_ROOT}/include
     -D BoostLib_LIBRARY_DIRS=${BOOST_ROOT}/lib
    -D Trilinos_ENABLE_HDF5=ON
      -D HDF5_INCLUDE_DIRS=${HDF5_ROOT}/include
      -D HDF5_LIBRARY_DIRS=${HDF5_ROOT}/lib
    -D Trilinos_ENABLE_Netcdf=ON
      -D Netcdf_INCLUDE_DIRS=${NETCDF_C_ROOT}/include
      -D Netcdf_LIBRARY_DIRS=${NETCDF_C_ROOT}/lib64

    -D TPL_ENABLE_BLAS:BOOL=ON
      -D BLAS_LIBRARY_DIRS="$MKLROOT/lib/intel64;$CMPROOT/lib"
      -D BLAS_LIBRARY_NAMES="pthread;mkl_intel_lp64;mkl_intel_thread;mkl_core"
    -D TPL_ENABLE_LAPACK:BOOL=ON
      -D LAPACK_LIBRARY_DIRS="$MKLROOT/lib/intel64"
      -D LAPACK_LIBRARY_NAMES="pthread;mkl_intel_lp64;mkl_intel_thread;mkl_core"
      -D Trilinos_EXTRA_LINK_FLAGS:STRING="-liomp5"
)

[jhux2]

We also have performance tests on amber:

module load aue/intel-oneapi-compilers/2024.1.0 aue/openmpi aue/cmake/3.29.4
module load aue/netcdf-c/4.9.2-oneapi-2024.2.1-intel-oneapi-mpi-2021.13.1 aue/parallel-netcdf/1.12.3-oneapi-2024.2.1-intel-oneapi-mpi-2021.13.1 aue/hdf5 aue/boost
# Swap back to openmpi for intel oneapi compilers
module load aue/openmpi/4.1.6-oneapi-2024.1.0

and they use these cmake options:

-DKokkos_ENABLE_ATOMICS_BYPASS=ON
-DKokkos_ARCH_SPR=ON
-DKokkos_ENABLE_SERIAL=ON

[jhux2]

Second, and just as complicated, I backed down to the TACC ICX partition (dual socket, 40 cores/socket), simulation time seem roughly the same speed as a legacy Stanford platform

Is this with Trilinos develop? How are you running (MPI ranks, threads), and could you please provide some continuity timing logs?

[spdomino]

Thank you for the reply.

Have your internal tests showed anything out of the ordinary lately by way of performance? Here, I should note that Aurora is not the same as this Intel CPU-based machine (Saphire Rapids). However, all I see by way of MKL specification is a few diffs. Would you think that could affect the AMG efficiency?

This is a "small" case by my old standards (roughly 30 million elements), however, a study that still requires a capability-class machine provided by the NSF (we have multiple cases to run). My three cases have been specifying the following:

#SBATCH -p icx 
#SBATCH -N 10 
#SBATCH --ntasks-per-node=80

#SBATCH -p skx 
#SBATCH -N 24 
#SBATCH --ntasks-per-node=40

#SBATCH -p spr
#SBATCH -N 10 
#SBATCH --ntasks-per-node=112

I also have this line floating around:

export OMP_NUM_THREADS=1

I have not timed Muelu yet, however, I can try at the expense of my allocation:) If you have no other suggestions, I can try that in the next week. If I see anything different between the most recent development and one from April 14th, 2026 (when I built on a Stanford platform) I will send that and we can proceed as you see best.

[spdomino]

@jhux2 - could you also please review the vintage "milestone.xml" file to see if things still look okay with a modern viewpoint perspective?

[jhux2]

@spdomino That file looks ok to me. As you know, the dropping threshold may be problem-dependent.

[spdomino]

Understood... This is an orthogonal mesh and, as such, exercising the edge-based (seven-point canonical stencil on Hex8).

ContinuityEQS 29 0.00398241 0.0785238 1

However, for around one second of a 3x3 momentum solve, I see 45 s of continuity - unlike anything I have ever seen since (dare I say) we were running SGS/GMRES for continuity (back in the Aztec/ML days) and taking 200 steps. Moreover, we are sitting on a CPU-based 35K elem/core balancing.

I guess I'll create a verbose run and report back. Also, for this data point, I reverted to my April 14th build on SPX.

[spdomino]

Hi @rcknaus. Could you look at our Trilinos config for a Sapphire rapids build and provide ant input? I don’t recall anything I did differently on previous builds. But it would be useful knowing what SHA1 you are currently using.

[rcknaus]

That build configuration looks standard to me. We're using 8e7286, 17.1, and haven't noticed a performance regression in the recent past, certainly nothing dramatic that'd cause 29 iterations to take 45 seconds with that node / core count. Anyway, nothing sticks out, but a verbose run might help like you say. You can also try https://github.com/kokkos/kokkos-tools kokkos-tools profiling to see what in particular is slow, to narrow things down.

[spdomino]

Thank you for the info. This is useful. On the SPR side, I found that Stampede3 has 1/2 the memory as the SNL SPR resource. The 30Mil case runs at 15 nodes (112 tasks per node; thread = 1). That places the elem/core count rather low (18k) and a bit expensive to run:)

The data on the ICX side (dual socket, 2x40 cores) paints a very clear picture that something is off in the CVFEM (element-based) setup. I ran a case for ten time steps, with recompute_preconditioner: no. CVFEM Muelu setup costs are 3000 s while for edge-based (EBVC) it is 57 s. Solver costs are 934 s vs 362 s, respectively. If I set reuse_preconditioner: yes, Muelu hangs. I am not sure why this is the case as we test this use case in the Nalu codes

The formal log files are attached for all cases, as is a small table. Any insight as to why CVFEM setup costs are 50x edge-based would be useful to know. Personally, never have seen this before. In the meantime, I'll try to revert back to the last 17.1 Trilinos snapshot. Sometimes keeping up with development on the Apps side makes this hard.

Discret.	System	Assemble	Solve	P-Setup	Recompute	Reuse
CVFEM	Mom	10	23	< 1	n/a	n/a
CVFEM	Cont	3	934	3000	FALSE	FALSE
EBVC	Mom	1	9	< 1	n/a	n/a
EBVC	Cont	< 1	361	57	FALSE	FALSE
EBVC	Mom	FAILED	FAILED	n/a	n/a	n/a
EBVC	Cont	FAILED	FAILED	hr+ at setup	FALSE	TRUE
CVFEM	Mom	FAILED	FAILED	n/a	n/a	n/a
CVFEM	Cont	FAILED	FAILED	hr+ at setup	FALSE	TRUE

mpf_K1h_600x400x100_0p001_Full_cvfem.logIcxVerbosity.txt
mpf_K1h_600x400x100_0p001_Full_cvfem.logIcxVerbosityReuse.txt
mpf_K1h_600x400x100_0p001_Full_ebvc.logIcxVerbosity.txt
mpf_K1h_600x400x100_0p001_Full_ebvc.logIcxVerbosityReuse.txt

[cgcgcg]

Some observations. From the first file:

 --------------------------------------------------------------------------------
  ---                            Multigrid Summary                             ---
  --------------------------------------------------------------------------------
  Scalar              = double
  Number of levels    = 10
  Operator complexity = 9.03
  Smoother complexity = 9.10
  Cycle type          = V
 
  level      rows        nnz  nnz/row  c ratio  procs
    0    26262960  705639012    26.87               800
    1     6975488  222952690    31.96     3.77      800
    2     1904295   68255563    35.84     3.66      800
    3      717532   36701958    51.15     2.65      717
    4      495648   74838834   150.99     1.45      494
    5      240003  628808203  2620.00     2.07      240
    6      116635 1292148343 11078.56     2.06      116
    7       72778 1901730724 26130.57     1.60       72
    8       36515 1018982181 27905.85     1.99       36
    9       20872  419800156 20113.08     1.75       20

• For a 3D problem, the coarseing is most likely way too slow. Unless the problem is really anisotropic the coarseing rate should be closer to 27.
• Starting with level 6 the matrices are pretty much dense.

This could explain why the setup is so slow. Not sure if this means that the matrix is bad or that something in MueLu broke.

[jhux2]

I agree with Christian's observations. I'm hard pressed to think of a case where we've seen 9 coarse grids (unless something is going wrong). His idea to switch to distance Laplacian is a good one.

[spdomino]

The results look sane [1], with the solver convergence in the 20-ish range. The simulation is simply extremely slow. I am also concerned over the lack of reuse being operational. The Nalu code base calls an internal Muelu “reuse” function.

I’ll try to look at the release Robert is using to see if my app code is too up to date.

[spdomino]

Oh. This is a high aspect ratio mesh. I’ll explore other options available on the app side.

[cgcgcg]

High aspect ratio mesh, but isotropic coefficients? You could try to use

<Parameter name="aggregation: drop scheme" type="string" value="distance laplacian"/>

[spdomino]

I think we moved to this xml since the previous bug report I field in April was again killed due to a bad memory allocation.

The edge case is handling this, and I checked to make sure the CVFEM was using a reduced LHS laplace operator.

I’ll revert to testing with the standard distance laplacian approach.

I’ll also test more on our local cases what might be going on with the MueluReuse call. Is that well tested?

https://github.com/COMERI/Mare-Nalu/blob/91bef83c35d3f2ae8a99ffc46353b6a0be4f874e/src/LinearSolver.C#L136