When I try to precompile the latest master (097ee11) on a machine which is usually reliable using the manual install instructions https://mabarnes.github.io/moment_kinetics/dev/manual_setup/, I now see this output
┌ CompositionsBase → CompositionsBaseInverseFunctionsExt
│ [pid 2781818] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ fs_event 0xcce120->0x7f36fd34cdf0
│ timer 0xe0f4f0->0x7f36fd34ce20
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└
┌ LoweredCodeUtils
│ WARNING: using IR.SSAValue in module LoweredCodeUtils conflicts with an existing identifier.
│ WARNING: using IR.SlotNumber in module LoweredCodeUtils conflicts with an existing identifier.
└
Is this caused by changes related to #273?
I have checked that the branch here https://github.com/mabarnes/moment_kinetics/commits/electric_field_switch/ without the timer changes does not have the first error message (the LoweredCodeUtils error is present). In this older branch, I can still use
julia --project -O3 precompile.jl
to make moment_kinetics.so and run
mpirun -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'
successfully. However, in the latest master, with the manual install instructions I get the following error when I run the above command
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
getting local rank failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
orte_ess_init failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
ompi_mpi_init: ompi_rte_init failed
--> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[tplxint4.nat.physics.ox.ac.uk:2777267] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
ERROR: LoadError: Failed to precompile Test [8dfed614-e22c-5e08-85e1-65c5234f0b40] to "*/excalibur/moment_kinetics_test_install/.julia/compiled/v1.10/Test/jl_D9dYor".
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
@ Base ./loading.jl:2468
[3] compilecache
@ ./loading.jl:2340 [inlined]
[4] (::Base.var"#968#969"{Base.PkgId})()
@ Base ./loading.jl:1974
[5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
@ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
[6] #mkpidlock#6
@ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
[7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
@ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
[8] #invokelatest#2
@ ./essentials.jl:894 [inlined]
[9] invokelatest
@ ./essentials.jl:889 [inlined]
[10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
@ Base ./loading.jl:2983
[11] maybe_cachefile_lock
@ ./loading.jl:2980 [inlined]
[12] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:1970
[13] __require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:1812
[14] #invoke_in_world#3
@ ./essentials.jl:926 [inlined]
[15] invoke_in_world
@ ./essentials.jl:923 [inlined]
[16] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:1803
[17] macro expansion
@ ./loading.jl:1790 [inlined]
[18] macro expansion
@ ./lock.jl:267 [inlined]
[19] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:1753
[20] #invoke_in_world#3
@ ./essentials.jl:926 [inlined]
[21] invoke_in_world
@ ./essentials.jl:923 [inlined]
[22] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1746
[23] include(mod::Module, _path::String)
@ Base ./Base.jl:495
[24] include(x::String)
@ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install/moment_kinetics/test/runtests.jl:1
[25] top-level scope
@ ~/excalibur/moment_kinetics_test_install/moment_kinetics/test/runtests.jl:3
[26] include(fname::String)
@ Base.MainInclude ./client.jl:489
[27] top-level scope
@ none:1
in expression starting at /*/moment_kinetics/test/setup.jl:10
in expression starting at /*/moment_kinetics/test/runtests.jl:1
[tplxint4:2777215] *** Process received signal ***
[tplxint4:2777215] Signal: Segmentation fault (11)
[tplxint4:2777215] Signal code: Address not mapped (1)
[tplxint4:2777215] Failing at address: 0x30
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[tplxint4:2777215] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f324a61a090]
[tplxint4:2777215] [ 1] /usr/lib/x86_64-linux-gnu/libpmix.so.2(pmix_ptl_base_lost_connection+0x9fe)[0x7f3247ddedbe]
[tplxint4:2777215] [ 2] /usr/lib/x86_64-linux-gnu/libpmix.so.2(pmix_ptl_base_recv_handler+0x189)[0x7f3247de0069]
[tplxint4:2777215] [ 3] /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x2113f)[0x7f324a7ea13f]
[tplxint4:2777215] [ 4] /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x52f)[0x7f324a7ea87f]
[tplxint4:2777215] [ 5] /usr/lib/x86_64-linux-gnu/libpmix.so.2(+0x73d56)[0x7f3247da3d56]
[tplxint4:2777215] [ 6] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f324a54d609]
[tplxint4:2777215] [ 7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f324a6f6353]
[tplxint4:2777215] *** End of error message ***
./activate_mk.sh: line 22: 2777215 Segmentation fault (core dumped) mpirun -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'
We may need to add StatsBase to https://mabarnes.github.io/moment_kinetics/dev/manual_setup/, e.g.,
$ julia --project -O3
> using Pkg; Pkg.add("StatsBase")
Unfortunately adding StatsBase did not fix the MPI error, although it did remove the precompile warning.
@johnomotani Does this make sense, or is something else going wrong here? I am trying to make a script for @MantasAbazorius to use to setup moment kinetics on the same machine, and precompile.jl might only be making a valid moment_kinetics.so when run from command line directly (i.e., not in a bash script).
When I try to precompile the latest master (097ee11) on a machine which is usually reliable using the manual install instructions https://mabarnes.github.io/moment_kinetics/dev/manual_setup/, I now see this output
Is this caused by changes related to #273?
I have checked that the branch here https://github.com/mabarnes/moment_kinetics/commits/electric_field_switch/ without the timer changes does not have the first error message (the
LoweredCodeUtilserror is present). In this older branch, I can still useto make
moment_kinetics.soand runsuccessfully. However, in the latest master, with the manual install instructions I get the following error when I run the above command
We may need to add
StatsBaseto https://mabarnes.github.io/moment_kinetics/dev/manual_setup/, e.g.,Unfortunately adding
StatsBasedid not fix the MPI error, although it did remove the precompile warning.@johnomotani Does this make sense, or is something else going wrong here? I am trying to make a script for @MantasAbazorius to use to setup moment kinetics on the same machine, and
precompile.jlmight only be making a validmoment_kinetics.sowhen run from command line directly (i.e., not in a bash script).