eCLM-PDAF crashes after some time for multivariate assimilation with PDAF-OMI

[s7yoewer]

The model crashes after some time for each simulation (something around 4,5 years). When restarting the model after 4 years with the given restart files, it works fine again until sth around 4.5 years. Sometimes it is a little bit longer and sometimes a little shorter but this is roughly the duration. The .err file only gives
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 000014ED66A45BF0 Unknown Unknown Unknown
libpscom.so.2.0.0 000014ED65D2C5A7 pscom_progress Unknown Unknown
libpscom.so.2.0.0 000014ED65D2CA03 pscom_wait_any Unknown Unknown
libmpi.so.12.3.1 000014ED68799C03 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED68754A61 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED68725A44 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED68725E93 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED68699ED5 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED6870F4CD Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED6870F654 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED6870F874 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED686992CE Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED6870E584 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED6870E645 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED6870E7A4 Unknown Unknown Unknown
libmpi.so.12.3.1 000014ED685E9423 PMPI_Barrier Unknown Unknown
libmpifort.so.12. 000014ED6715898C pmpi_barrier Unknown Unknown
tsmp-pdaf 0000000000423970 Unknown Unknown Unknown
tsmp-pdaf 0000000000419D6D Unknown Unknown Unknown
libc.so.6 000014ED66A305D0 Unknown Unknown Unknown
libc.so.6 000014ED66A30680 __libc_start_main Unknown Unknown
tsmp-pdaf 0000000000419C85 Unknown Unknown Unknown

so I am not sure why this happens. @jjokella do you have an idea and have already seen sth similar. Given the restart error in eCLM, I can only use the first years of my simulation to compare to some OL. This does not happen when no observations are assimilated (for an OL run), so I subspect some memory issues (?).

[jjokella]

Hm, this sounds like an issue that will be hard to track. I have seen memory issues, but cannot currently recall something completely related. Nevertheless, some suggestions:

• checking that it is a memory issue (possibly some de-allocation missing): Check the run on https://www.fz-juelich.de/en/jsc/services/user-support/software-tools/llview in particular the memory usage graphs over time

• All the Unknown in the traceback suggest that you could try running the same simulation with a debug build (costs a lot of c-h of course, but would deliver a part in the code where it crashes).

• quick-fix for longer simulations: regarding the restart issue, this should be resolved by eCLM-PDAF: Set process-specific tag in coupler restart file name eCLM#91

[s7yoewer]

The memory is constant in the first hours. Then, it suddenly starts to rise.

I can try to rerun with the debug build to at least know where the simulation crashes. But I am quite confused that the memory usage suddenly gets larger and larger after a lot observations were already assimilated.

[s7yoewer]

Something weird is also happening with the CPUs

[s7yoewer]

With the restart thing, I mean that restarting eCLM leads to some changes in the soil water content in the top layer (HPSCTerrSys/eCLM#90)

[jjokella]

The memory is constant in the first hours. Then, it suddenly starts to rise.

This is strange... Can you confirm, whether the simulation is already running in these first hours. Or could all this time even be initialization in such a large run?

Could be interesting to check the llview output from the OL against this.

All in all, it seems to be a memory issue. We can bring it up in tomorrow's meeting(s).

[s7yoewer]

The first history files are written at 5:34 and the first assimilation time step is at 5:43, so the initialization works fine.

The OL looks fine, the memory and CPUS have quite the constant load.

I will try with DEBUG output enabled.

[jjokella]

The first history files are written at 5:34 and the first assimilation time step is at 5:43, so the initialization works fine.

Ok, so data assimilation updates happening for five hours before the strange behavior starts if I understand correctly. That makes it even stranger...