Hi all,
I am trying to perform simulations on the Discoverer cluster ( https://docs.discoverer.bg/index.html) using the latest Einstein Toolkit (ETK) release (ET_2022_05) and the Spritz GRMHD code. To compile ETK on Discoverer, I am attaching the simfactory configuration files which I had newly prepared. I am also attaching the list of modules which were loaded.
To submit the simulation, for instance, I use the following simfactory command:
sim submit BNS_IF_fluxCT_dx018_q10_RPA_RotGas_E8e49 --parfile=./par/BNS_IF_fluxCT_dx018_q10_RPA_RotGas_E8e49.par --config=newspritzgnu --machine=discoverer --procs=256 --num-threads=1 --ppn-used=128 --walltime=24:00:00
Unfortunately, my simulations crash after running for some time. My guess is that I might not be correctly setting the configuration options or flags during compilation, which might affect my simulation during runtime, but I am not completely certain. I am attaching the output file, the error file, the parfile as well as the generated backtrace for the simulation which used 256 procs. I also looked at the hexadecimal addresses in the backtrace with addr2line, but unfortunately all of them return "??:0"
I also noticed that when changing the number of processors, the simulation crashes at different times. But if I keep the number of processors as the same, the simulation always crashes at the same point. For instance, simulation with 256 processors ran for about 2 hours on the cluster, and crashed after completing about 4600 iterations. One submitted with 1280 processors ran for about 12 hours and crashed after completing about 32000 iterations. Simulation with 1792 processors instead crashed soon after the start of the simulation (within few minutes), even before reaching iteration 0. For all cases, I always set number of threads as 1.
If you have any suggestions or insights on why the simulations crash and in case I have any incorrect settings in the configuration files, kindly let me know. I would greatly appreciate your help. If you need any further information from my side, please let me know too.
Thank you very much. Kind regards, Jay Kalinani