The library 2decomp is a Fortran library compatible with the Fortran 2008 standard. It requires a MPI library compatible with MPI-2.0 with extended Fortran support. The following optional libraries can be used :
- ADIOS2, version 2.9.0 was tested
- FFTW3, version 3.3.10 was tested
- Intel oneMKL (oneAPI Math Kernel Library), version 2023.0.0 was tested
- Nvidia GPU-related libraries, NVHPC version 22.7 and CUDA version 11.8 were tested
- Caliper, version 2.9.1 was tested
The build system is driven by cmake. It is good practice to directly point to the
MPI Fortran wrapper that you would like to use to guarantee consistency between Fortran compiler and MPI.
This can be done by setting the default Fortran environmental variable
$ export FC=my_mpif90
To generate the build system run
$ cmake -S $path_to_sources -B $path_to_build_directory -DOPTION1 -DOPTION2 ...
If the directory does not exist it will be generated and it will contain the configuration files.
The configuration can be further
edited by using the ccmake utility as
$ ccmake $path_to_build_directory
and editing as desired, variables that are likely of interest are: CMAKE_BUILD_TYPE and FFT_Choice;
additional variables can be shown by entering "advanced mode" by pressing t.
By default a RELEASE build will built, other options for CMAKE_BUILD_TYPE are DEBUG and DEV which
turn on debugging flags and additionally try to catch coding errors at compile time, respectively.
The behaviour of debug and development versions of the library can be changed before the
initialization using the variable decomp_debug or the environment variable DECOMP_2D_DEBUG.
The value provided with the environment variable must be a positive integer below 9999.
Two BUILD_TARGETS are available namely mpi and gpu. For the mpi target no additional
options should be required. whereas for gpu extra options are necessary at the configure stage.
Please see section GPU Compilation
Once the build system has been configured, you can build 2DECOMP&FFT` by running
$ cmake --build $path_to_build_directory -j <nproc>
appending -v will display additional information about the build, such as compiler flags.
After building the library can be tested. Please see section Testing and examples
Options can be added to change the level of verbosity. Finally, the build library can be installed by running
$ cmake --install $path_to_build_directory
The default location for libdecomp2d.a is $path_to_build_directory/opt/lib
or $path_to_build_directory/opt/lib64 unless the variable CMAKE_INSTALL_PREFIX is modified.
The module files generated by the build process will similarly be installed to
$path_to_build_directory/opt/install,
users of the library should add this to the include paths for their program.
As indicated above, by default a static libdecomp2d.a will be compiled,
if desired a shared library can be built by setting BUILD_SHARED_LIBS=ON either on the command line:
$ cmake -S $path_to_sources -B $path_to_build_directory -DBUILD_SHARED_LIBS=ON
or by editing the configuration using ccmake.
This might be useful for a centralised install supporting multiple users that is upgraded over time.
Occasionally a clean build is required, this can be performed by running
$ cmake --build $path_to_build_directory --target clean
The library can perform multi GPU offoloading using the NVHPC compiler suite for NVIDIA hardware. The implementation is based on CUDA-aware MPI and NVIDIA Collective Communication Library (NCCL). The FFT is based on cuFFT.
To properly configure for GPU build the following needs to be used
$ cmake -S $path_to_sources -B $path_to_build_directory -DBUILD_TARGET=gpu
Note, further configuration can be performed using ccmake, however the initial configuration of GPU builds must include the -DBUILD_TARGET=gpu flag as shown above.
By default CUDA aware MPI will be used together with cuFFT for the FFT library. The configure will automatically look for the GPU architecture available on the system. If you are building on a HPC system please use a computing node for the installation. Useful variables to be added are
-DENABLE_NCCL=yesto activate the NCCL collectives-DENABLE_MANAGED=yesto activate the automatic memory management form the NVHPC compiler If you are getting the following error
-- The CUDA compiler identification is unknown
CMake Error at /usr/share/cmake/Modules/CMakeDetermineCUDACompiler.cmake:633 (message):
Failed to detect a default CUDA architecture.
It is possible that your default C compiler is too recent and not supported by nvcc . You might be able to solve the issue by adding
-DCMAKE_CUDA_HOST_COMPILER=$supported_gcc
At the moment the supported CUDA host compilers are gcc11 and earlier.
When building a code that links 2DECOMP&FFT using a Makefile you will need to add the include and link paths as appropriate (inlude/ and lib/ under the installation directory, respectively).
DECOMP_ROOT = /path/to/2decomp-fft
DECOMP_BUILD_DIR = $(DECOMP_ROOT)/build
DECOMP_INSTALL_DIR ?= $(DECOMP_BUILD_DIR)/opt # Use default unless set by user
INC += -I$(DECOMP_INSTALL_DIR)/include
# Users build/link targets
LIBS = -L$(DECOMP_INSTALL_DIR)/lib64 -L$(DECOMP_INSTALL_DIR)/lib -ldecomp2d
OBJ = my_exec.o
my_exec: $(OBJ)
$(F90) -o $@ $(OBJ) $(LIBS)
In case 2DECOMP&FFT has been compiled with an external FFT, such as FFTW3, LIBS
should also contain the following
FFTW3_PATH=/my_path_to_FFTW/lib
LIBFFT=-L$(FFTW3_PATH) -lfftw3 -lfftw3f
LIBS += $(LIBFFT)
In case of 2DECOMP&FFT compiled for GPU with NVHPC, linking against cuFFT is mandatory
LIBS += -cudalib=cufft
In case of NCCL the following is required
LIBS += -cudalib=cufft,nccl
It is also possible to drive the build and installation of 2decomp-fft from a Makefile such as in the following example code
FC = mpif90
BUILD = Release
DECOMP_ROOT = /path/to/2decomp-fft
DECOMP_BUILD_DIR = $(DECOMP_ROOT)/build
DECOMP_INSTALL_DIR ?= $(DECOMP_BUILD_DIR)/opt # Use default unless set by user
INC += -I$(DECOMP_INSTALL_DIR)/include
# Users build/link targets
LIBS = -L$(DECOMP_INSTALL_DIR)/lib64 -L$(DECOMP_INSTALL_DIR)/lib -ldecomp2d
# Building libdecomp.a
$(DECOMP_INSTALL_DIR)/lib/libdecomp.a:
FC=$(FC) cmake -S $(DECOMP_ROOT) -B $(DECOMP_BUILD_DIR) -DCMAKE_BUILD_TYPE=$(BUILD) -DCMAKE_INSTALL_PREFIX=$(DECOMP_INSTALL_DIR)
cmake --build $(DECOMP_BUILD_DIR) --target decomp2d
cmake --build $(DECOMP_BUILD_DIR) --target install
# Clean libdecomp.a
clean-decomp:
cmake --build $(DECOMP_BUILD_DIR) --target clean
rm -f $(DECOMP_INSTALL_DIR)/lib/libdecomp.a
Profiling can be activated via cmake configuration,
the recommended approach is to run the initial configuration as follows:
$ export caliper_DIR=/path/to/caliper/install/share/cmake/caliper
$ export CXX=mpicxx
$ cmake -S $path_to_sources -B $path_to_build_directory -DENABLE_PROFILER=caliper
where ENABLE_PROFILER is set to the profiling tool desired, currently supported values are: caliper.
Note that when using caliper a C++ compiler is required as indicated in the above command line.
In order to compile with the MKL libraries the following environmental variable needs to be set up
$ export MKL_DIR=${MKLROOT}/lib/cmake/mkl
and select the MKL backend by setting FFT_Choice=mkl.
To use the new IntelLLVM compiler, up until the 2023 version, specify it as the Fortran compiler using
export export FC="mpiifort -fc=ifx"
and when building with ADIOS2 support you must also specify the C and CXX compilers
export CXX="mpiicpc -cxx=icpx"
export CC="mpiicc -cc=icx"
From the 2024 version new MPI wrapper are available as mpiifx, mpiicx and mpiicpx.
This variable is automatically added in builds with the adios2 IO backend.
This variable is automatically added in debug and dev builds. Extra information is printed when it is present.
When this variable is not present, the library uses single precision. When it is present, the library uses double precision. This preprocessor variable is driven by the CMake on/off variable DOUBLE_PRECISION.
This variable is valid for double precision builds only. When it is present, snapshots are written in single precision. This preprocessor variable is driven by the CMake on/off variable SINGLE_PRECISION_OUTPUT.
This variable is automatically added when selecting the profiler. It activates the profiling sections of the code.
This preprocessor variable is not valid for GPU builds. It leads to padded alltoall operations. This preprocessor variable is driven by the CMake on/off variable EVEN.
This variable leads to overwrite the input array when computing FFT. The support of this flag does not always correspond to in-place transforms, depending on the FFT backend selected, as described above. This preprocessor variable is driven by the CMake on/off variable ENABLE_INPLACE.
This variable is used to debug the halo operations. This preprocessor variable is driven by the CMake on/off variable HALO_DEBUG.
This variable is used in the example used to test the halo operations. If the variable is defined, the test are using arrays defined with the key opt_global set to .true.. Otherwise, the arrays are defined with the key set to .false..
This variable is automatically added in GPU builds.
This variable is valid only for GPU builds. The NVIDIA Collective Communication Library (NCCL) implements multi-GPU and multi-node communication primitives optimized for NVIDIA GPUs and Networking.
The library adios2 can be used as a backend for IO. The version 2.9.0 was tested, is supported and can be downloaded here. Below are build instructions for the library. However, it is recommended to use the one provided by the administrators of the computing centre if available.
$ wget https://github.com/ornladios/ADIOS2/archive/refs/tags/v2.9.0.tar.gz
$ tar xzf v2.9.0.tar.gz
$ mkdir 2.9.0_tmp && cd 2.9.0_tmp
$ CC=mpicc CXX=mpicxx FC=mpif90 cmake -S ../ADIOS2-2.9.0 -DCMAKE_INSTALL_PREFIX=../2.9.0_bld
$ make -j
$ make -j test
$ make -j install
To build 2DECOMP&FFT with the adios2 IO backend, one can provide the package configuration for adios2
in the PKG_CONFIG_PATH environment variable,
this should be found under /path/to/adios2/install/lib/cmake/adios2.
One can also provide the option -Dadios2_DIR=/path/to/adios2/install/lib/cmake/adios2.
Then either specify on the command line when configuring the build
$ cmake -S . -B ./build -DIO_BACKEND=adios2 -Dadios2_DIR=/path/to/adios2/install/lib/cmake/adios2
or modify the build configuration using ccmake. Please note that the support for ADIOS2 is not complete. Currently, for a given IO operation, when the ADIOS2 backend is not supported, the MPI backend is used.
The library fftw can be used as a backend for the FFT engine.
The version 3.3.10 was tested, is supported and can be downloaded
here.
Please note that one should build fftw and decomp2d against the same compilers.
For build instructions, please check here.
Below is a suggestion for the compilation of the library in double precision
(add --enable-single for a single precision build):
$ wget http://www.fftw.org/fftw-3.3.10.tar.gz
$ tar xzf fftw-3.3.10.tar.gz
$ mkdir fftw-3.3.10_tmp && cd fftw-3.3.10_tmp
$ ../fftw-3.3.10/configure --prefix=xxxxxxx/fftw3/fftw-3.3.10_bld --enable-shared
$ make -j
$ make -j check
$ make install
Please note that the resulting build is not compatible with cmake
(FFTW/fftw3#130).
As a workaround, one can open the file
/path/to/fftw3/install/lib/cmake/fftw3/FFTW3Config.cmake
and comment the line
include ("${CMAKE_CURRENT_LIST_DIR}/FFTW3LibraryDepends.cmake")
To build 2decomp&fft against fftw3, one can provide the package configuration for fftw3
in the PKG_CONFIG_PATH environment variable,
this should be found under /path/to/fftw3/install/lib/pkgconfig.
One can also provide the option -DFFTW_ROOT=/path/to/fftw3/install.
Then either specify on the command line when configuring the build
$ cmake -S . -B build -DFFT_Choice=<fftw|fftw_f03> -DFFTW_ROOT=/path/to/fftw3/install
or modify the build configuration using ccmake.
Note the legacy fftw interface lacks interface definitions
and will fail when stricter compilation flags are used (e.g. when -DCMAKE_BUILD_TYPE=Dev)
for this it is recommended to use fftw_f03 which provides proper interfaces.
The library caliper can be used to profile the execution of the code. The version 2.9.1 was tested and is supported, version 2.8.0 has also been tested and is still expected to work. Please note that one must build caliper and decomp2d against the same C/C++/Fortran compilers and MPI libray. For build instructions, please check here and here. Below is a suggestion for the compilation of the library using the GNU compilers:
$ git clone https://github.com/LLNL/Caliper.git caliper_github
$ cd caliper_github
$ git checkout v2.9.1
$ mkdir build && cd build
$ cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_INSTALL_PREFIX=../../caliper_build_2.9.1 -DWITH_FORTRAN=yes -DWITH_MPI=yes -DBUILD_TESTING=yes ../
$ make -j
$ make test
$ make install
After installing Caliper ensure to set caliper_DIR=/path/to/caliper/install/share/cmake/caliper.
Following this the 2decomp-fft build can be configured to use Caliper profiling as
$ cmake -S . -B -DENABLE_PROFILER=caliper
or by modifying the configuration to set ENABLE_PROFILER=caliper via ccmake.