Hello,
Some network configurations were recently changed on SDSC's
Expanse and I wanted to update the Simfactory entry to add an env
variable (as recommended by XSEDE's help desk).
I did so and ran the tests and found numerous failures. According to an
expanse_2_64.log file I have on my computer, these tests did not fail in
the past. The tests fail only for 2 MPI processes.
An example of a test that fails is rnsA2, and this is the the tail of the log
file:
INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 2 processes
WARNING level 0 from host exp-4-26.expanse.sdsc.edu process 0
in thorn Carpet, file /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148:
-> The environment variable CACTUS_NUM_PROCS is set to 1, but there are 2 MPI processes. This may indicate a severe problem with the MPI startup mechanism.
Rank 0 with PID 1194507 received signal 6
cactus_sim: /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.
Writing backtrace to rnsA2/backtrace.0.txt
Rank 1 with PID 1194508 received signal 6
Writing backtrace to rnsA2/backtrace.1.txt
srun: error: exp-4-26: tasks 0-1: Aborted (core dumped)
IBRUN: launch command: srun -n 2 --ntasks-per-node 2 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim -L 3 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/output-0000/arrangements/EinsteinInitialData/Hydro_RNSID/test/rnsA2.par
IBRUN: MPI job exited with code: 134
Other tests behave similarly, e.g. Vaidya2:
INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 2 processes
WARNING level 0 from host exp-4-26.expanse.sdsc.edu process 0
in thorn Carpet, file /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148:
-> The environment variable CACTUS_NUM_PROCS is set to 1, but there are 2 MPI processes. This may indicate a severe problem with the MPI startup mechanism.
Rank 0 with PID 1183519 received signal 6
Writing backtrace to Vaidya2/backtrace.0.txt
cactus_sim: /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed.
Rank 1 with PID 1183520 received signal 6
Writing backtrace to Vaidya2/backtrace.1.txt
srun: error: exp-4-26: tasks 0-1: Aborted (core dumped)
IBRUN: launch command: srun -n 2 --ntasks-per-node 2 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim -L 3 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/output-0000/arrangements/EinsteinExact/EinsteinExact_Test/test/Vaidya2.par
IBRUN: MPI job exited with code: 134
Given that I see in the testsuite_results repo the same failing tests
(as run by Roland), I can exclude that the new env variable that I
added is the reason for the failures.
Any idea of what is going on?
Thanks,
Gabriele