Multi-process ParFlow stand-alone fails on GNU stack

[mvhulten]

When compiling ParFlow with the Intel compiler (default), I can successfully run it manually (through tclsh).
However, when compiling ParFlow with GCC and OpenMPI, running fails with

There are not enough slots available in the system to satisfy the 2
slots that were requested by the application:
…

I was sure to use the same environment file before running ParFlow.

• Intel case: ParaStationMPI/5.10.0-1 is loaded
• GNU case: OpenMPI/5.0.5 is loaded

I'm using JURECA and Stages/2025.

[mvhulten]

It fails at the line pfrun default_single.

I double-checked that in the same environment (both GNU and Intel) with GCC- and Intel/LLVM-compiled binaries of a trivial hello-world program works for both compilers.

[kvrigor]

@mvhulten Do I understand correctly that pfrun default_single passes on Intel+ParaStationMPI but fails on GCC+OpenMPI? If so, can this issue be reproduced on other machines, e.g. Marvin, JUWELS, local Ubuntu machine, etc.?

[mvhulten]

Yes, but only when I'm submitting it through Slurm (in an interactive job). Otherwise, it is the other way around, I found out. These are maybe two different issues and need some details...

For the GCC+OpenMPI case,

vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3  # make sure that SLURM_JOBID is not set at this point

default_single : PASSED
vanhulten1@jrlogin09:.../parflow/test/tcl$ salloc --partition=dc-cpu-devel --nodes=1 --account=cdetect --time=0:10:00
salloc: Granted job allocation 13986807
salloc: Waiting for resource configuration
salloc: Nodes jrc0096 are ready for job
vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 6
slots that were requested by the application:

  /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/parflow

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRRTE processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------
child process exited abnormally
    while executing
"exec sh /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/run default_single 6"
    ("eval" body line 1)
    invoked from within
"eval exec sh $Parflow::PARFLOW_DIR/bin/run  $runname $NumProcs"
    invoked from within
"if [pfexists Process.Command] {
        set command [pfget Process.Command]
        puts [format "Using command : %s" [format $command $NumProcs $runname]]
        puts [e..."
    (procedure "pfrun" line 53)
    invoked from within
"pfrun default_single"
    (file "default_single.tcl" line 324)

For the Intel+ParaStationMPI case,

vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3
srun: error: job_submit_filter: please specify the job's account with -A or --account option
srun: error: Hint: to find available budget accounts from each project please run: jutil user projects
srun: error: Unable to allocate resources: Invalid account or account/partition combination specified
child process exited abnormally
    while executing
"exec sh /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025Intel/bin/JURECADC_ParFlow/bin/run default_single 6"
    ("eval" body line 1)
    invoked from within
"eval exec sh $Parflow::PARFLOW_DIR/bin/run  $runname $NumProcs"
    invoked from within
"if [pfexists Process.Command] {
        set command [pfget Process.Command]
        puts [format "Using command : %s" [format $command $NumProcs $runname]]
        puts [e..."
    (procedure "pfrun" line 53)
    invoked from within
"pfrun default_single"
    (file "default_single.tcl" line 324)
vanhulten1@jrlogin09:.../parflow/test/tcl$ salloc --partition=dc-cpu-devel --nodes=1 --account=cdetect --time=0:10:00
salloc: Granted job allocation 13986806
salloc: Waiting for resource configuration
salloc: Nodes jrc0096 are ready for job
vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3

default_single : PASSED

The results are the same when running gmake test from the ParFlow build directory ($TSMP2_DIR/bld/JURECADC_ParFlow/ParFlow/src/ParFlow-build/): starting from Test 9, Intel fails on compute and GNU fails on login node.

[mvhulten]

After some testing, here are my findings and hypotheses.

In the case of GCC+OpenMPI, there is a binary mpirun available, which is taken (precedence over srun) on the login node when invoking tclsh default_single.tcl 1 2 3. This works fine.
Then when making a reservation, mpirun is of course also available, but this is not the right way to run jobs. Instead, srun should be used on JSC machines.
In the case of Intel+ParaStationMPI, there is no mpirun, just /usr/bin/srun, so the job fails without a reservation (srun is part of Slurm and so a reservation must be made) and with a reservation it succeeds.

For the GCC+OpenMPI case, I tried adding cmake_conf+=" -DMPIEXEC=srun -DMPIEXEC_NUMPROC_FLAG=-n" and setting SLURM_NTASKS=6, but it does not fix the problem.
When I set at runtime PARFLOW_MPIEXEC_EXTRA_FLAGS="--oversubscribe", tclsh default_single.tcl 1 2 3 takes longer than a few minutes (it hangs).

Likely shell variables (like PARFLOW_MPIEXEC_EXTRA_FLAGS and SLURM_NTASKS) are overridden by command parameters in the ParFlow runtime system (e.g. in a Tcl module).

[kvrigor]

FYI, ParFlow MPI options are baked in BuildParFlow.cmake:

TSMP2/cmake/BuildParFlow.cmake

Lines 80 to 82 in 0252d09

	-DMPIEXEC_EXECUTABLE=${MPIEXEC_EXECUTABLE}
	-DMPIEXEC_NUMPROC_FLAG=${MPIEXEC_NUMPROC_FLAG}
	-DPARFLOW_ENABLE_SLURM=${ENABLE_SLURM}

Can you try to play with these settings Marco? You can also log their values to see how the MPI options change when choosing different toolchains. These options may have to be fixed to account for JSC's srun as you rightly mentioned.

[mvhulten]

I though, let's set -DMPIEXEC_EXECUTABLE=srun, but

…/bld/JURECADC_ParFlow/ParFlow/src/ParFlow-build/srun: No such file or directory

MPIEXEC_EXECUTABLE clearly is not what I thought it was. I then set

-DMPIEXEC=srun
-DMPIEXEC_NUMPROC_FLAG=-n

but that turned up the same error, just like setting SLURM_NTASKS does not help. I think it is overridden somewhere in the scripts. I have to dig, or maybe @skollet could help?

The problem is that no more than 1 slot is available. This should illustrate it:

vanhulten1@jrlogin09:.../parflow/test/tcl$ salloc --partition=dc-cpu-devel --nodes=1 --account=cdetect --time=0:10:00
salloc: Granted job allocation 13993235
salloc: Waiting for resource configuration
salloc: Nodes jrc0736 are ready for job
vanhulten1@jrlogin09:.../parflow/test/tcl$ export SLURM_NTASKS=6
vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 1 1

default_single : PASSED
vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 6
slots that were requested by the application:

  /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/parflow

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRRTE processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------
child process exited abnormally
    while executing
"exec sh /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/run default_single 6"
    ("eval" body line 1)
    invoked from within
"eval exec sh $Parflow::PARFLOW_DIR/bin/run  $runname $NumProcs"
    invoked from within
"if [pfexists Process.Command] {
        set command [pfget Process.Command]
        puts [format "Using command : %s" [format $command $NumProcs $runname]]
        puts [e..."
    (procedure "pfrun" line 53)
    invoked from within
"pfrun default_single"
    (file "default_single.tcl" line 324) 

[mvhulten]

I got a different error, which gave me a hint:

No PMIx server was reachable, but a PMI1/2 was detected.

It seems that all this time during my GCC+OpenMPI testing, SLURM_MPI_TYPE was not set. If this is set to pspmix, the job works. Loading the OpenMPI module also sets this variable. I did not think it was an environmental variable, because the same modules are loaded before and after sourcing the environment file. Of course, shell variables can be changed after loading modules, but I do not know why specifically this variable was unset.

It may be wise to alway reload your modules (sourcing an env file) after starting an interactive session (or normal job submission, where you probably really have to do this).

Moreover, I noticed that when running salloc and letting Slurm revoke your allocation, you stay in a subshell (which I found out when diffing set output and noticing a SHLVL that tracks shell nesting).

I am checking if this was the only issue, or if something from above was also needed...

...yes, CMake configuration changes were needed and so this note is a side remark. Patch will follow...