Hello,
Some network configurations were recently changed on SDSC's Expanse and I wanted to update the Simfactory entry to add an env variable (as recommended by XSEDE's help desk).
I did so and ran the tests and found numerous failures. According to an expanse_2_64.log file I have on my computer, these tests did not fail in the past. The tests fail only for 2 MPI processes.
An example of a test that fails is rnsA2, and this is the the tail of the log file:
INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 2 processes WARNING level 0 from host exp-4-26.expanse.sdsc.edu process 0 in thorn Carpet, file /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there are 2 MPI processes. This may indicate a severe problem with the MPI startup mechanism. Rank 0 with PID 1194507 received signal 6 cactus_sim: /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Writing backtrace to rnsA2/backtrace.0.txt Rank 1 with PID 1194508 received signal 6 Writing backtrace to rnsA2/backtrace.1.txt srun: error: exp-4-26: tasks 0-1: Aborted (core dumped) IBRUN: launch command: srun -n 2 --ntasks-per-node 2 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim -L 3 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/output-0000/arrangements/EinsteinInitialData/Hydro_RNSID/test/rnsA2.par
IBRUN: MPI job exited with code: 134
Other tests behave similarly, e.g. Vaidya2:
INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 2 processes WARNING level 0 from host exp-4-26.expanse.sdsc.edu process 0 in thorn Carpet, file /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there are 2 MPI processes. This may indicate a severe problem with the MPI startup mechanism. Rank 0 with PID 1183519 received signal 6 Writing backtrace to Vaidya2/backtrace.0.txt cactus_sim: /home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Rank 1 with PID 1183520 received signal 6 Writing backtrace to Vaidya2/backtrace.1.txt srun: error: exp-4-26: tasks 0-1: Aborted (core dumped) IBRUN: launch command: srun -n 2 --ntasks-per-node 2 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim -L 3 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/output-0000/arrangements/EinsteinExact/EinsteinExact_Test/test/Vaidya2.par
IBRUN: MPI job exited with code: 134
Given that I see in the testsuite_results repo the same failing tests (as run by Roland), I can exclude that the new env variable that I added is the reason for the failures.
Any idea of what is going on?
Thanks, Gabriele
Hello Gabriele,
hmm.
/home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there are 2 MPI processes. This may indicate a severe problem with the MPI startup mechanism.
IBRUN: launch command: srun -n 2 --ntasks-per-node 2 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim
Looking at these, I would have expected that CACTUS_NUM_PROCS is set to 2 given that -n is 2 (being the number of MPI ranks).
The current submitscript uses ibrun though current documentation uses srun. Maybe changing to srun helps? Though the srun command does seem to have 2 MPI procs in the way you expect to.
Can you check (in the RunScript in simulations/foo/output-0000/SIMFACTORY) what CACTUS_NUM_PROCS is set to?
If this works with "regular" runs but fails with the testsuite using --testsuite then the issue is most likely related to the complicated method simfactory has to use to set the number of MPI ranks.
I would check if the failing test is actually runnable only on 1 MPI rank (set in test.ccl). In that case, Cactus will try to run in it in a 2 MPI rank test suite but use only 1 MPI rank. Possibly ibrun ignores Cactus' request and uses only information provided by SLURM.
Yours, Roland
Hello Gabriele,
ok, I can at least partially answer this. Indeed RNS's A2 test is code to use only 1 MPI rank:
TEST rnsA2 { PROCS 1 }
and thus the most likely reason is that ibrun just pulls the number of MPI ranks from SLURM rather than from whatever simfactory tries to use.
Since ibrun is no longer documented on the SDSC page (at least I do not see it on https://www.sdsc.edu/support/user_guides/expanse.html), maybe the easiest fix is to remove it and use the srun command they document now?
Yours, Roland
Hello Gabriele,
hmm.
/home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there are 2 MPI processes. This may indicate a severe problem with the MPI startup mechanism.
IBRUN: launch command: srun -n 2 --ntasks-per-node 2 /expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim
Looking at these, I would have expected that CACTUS_NUM_PROCS is set to 2 given that -n is 2 (being the number of MPI ranks).
The current submitscript uses ibrun though current documentation uses srun. Maybe changing to srun helps? Though the srun command does seem to have 2 MPI procs in the way you expect to.
Can you check (in the RunScript in simulations/foo/output-0000/SIMFACTORY) what CACTUS_NUM_PROCS is set to?
If this works with "regular" runs but fails with the testsuite using --testsuite then the issue is most likely related to the complicated method simfactory has to use to set the number of MPI ranks.
I would check if the failing test is actually runnable only on 1 MPI rank (set in test.ccl). In that case, Cactus will try to run in it in a 2 MPI rank test suite but use only 1 MPI rank. Possibly ibrun ignores Cactus' request and uses only information provided by SLURM.
Yours, Roland
Hi Roland,
That sounds reasonable. I think I was originally using srun, but was recommended to move to ibrun. I will try with srun to see if it works, in which case I will update the simfactory entry and the testsuite results.
Gabrieel
On Thu, Jun 2, 2022 at 8:03 AM Roland Haas rhaas@illinois.edu wrote:
Hello Gabriele,
ok, I can at least partially answer this. Indeed RNS's A2 test is code to use only 1 MPI rank:
TEST rnsA2 { PROCS 1 }
and thus the most likely reason is that ibrun just pulls the number of MPI ranks from SLURM rather than from whatever simfactory tries to use.
Since ibrun is no longer documented on the SDSC page (at least I do not see it on https://www.sdsc.edu/support/user_guides/expanse.html), maybe the easiest fix is to remove it and use the srun command they document now?
Yours, Roland
Hello Gabriele,
hmm.
/home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there
are 2
MPI processes. This may indicate a severe problem with the MPI startup mechanism.
IBRUN: launch command: srun -n 2 --ntasks-per-node 2
/expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim
Looking at these, I would have expected that CACTUS_NUM_PROCS is set to 2 given that -n is 2 (being the number of MPI ranks).
The current submitscript uses ibrun though current documentation uses srun. Maybe changing to srun helps? Though the srun command does seem to have 2 MPI procs in the way you expect to.
Can you check (in the RunScript in simulations/foo/output-0000/SIMFACTORY) what CACTUS_NUM_PROCS is set to?
If this works with "regular" runs but fails with the testsuite using --testsuite then the issue is most likely related to the complicated method simfactory has to use to set the number of MPI ranks.
I would check if the failing test is actually runnable only on 1 MPI rank (set in test.ccl). In that case, Cactus will try to run in it in a 2 MPI rank test suite but use only 1 MPI rank. Possibly ibrun ignores Cactus' request and uses only information provided by SLURM.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello,
Reverting to srun fixes the problem. I updated the master branches for the testsuite results and simfactory.
Gabriele
On Thu, Jun 2, 2022 at 9:12 AM Gabriele Bozzola bozzola.gabriele@gmail.com wrote:
Hi Roland,
That sounds reasonable. I think I was originally using srun, but was recommended to move to ibrun. I will try with srun to see if it works, in which case I will update the simfactory entry and the testsuite results.
Gabrieel
On Thu, Jun 2, 2022 at 8:03 AM Roland Haas rhaas@illinois.edu wrote:
Hello Gabriele,
ok, I can at least partially answer this. Indeed RNS's A2 test is code to use only 1 MPI rank:
TEST rnsA2 { PROCS 1 }
and thus the most likely reason is that ibrun just pulls the number of MPI ranks from SLURM rather than from whatever simfactory tries to use.
Since ibrun is no longer documented on the SDSC page (at least I do not see it on https://www.sdsc.edu/support/user_guides/expanse.html), maybe the easiest fix is to remove it and use the srun command they document now?
Yours, Roland
Hello Gabriele,
hmm.
/home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there
are 2
MPI processes. This may indicate a severe problem with the MPI startup mechanism.
IBRUN: launch command: srun -n 2 --ntasks-per-node 2
/expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim
Looking at these, I would have expected that CACTUS_NUM_PROCS is set to 2 given that -n is 2 (being the number of MPI ranks).
The current submitscript uses ibrun though current documentation uses srun. Maybe changing to srun helps? Though the srun command does seem to have 2 MPI procs in the way you expect to.
Can you check (in the RunScript in simulations/foo/output-0000/SIMFACTORY) what CACTUS_NUM_PROCS is set to?
If this works with "regular" runs but fails with the testsuite using --testsuite then the issue is most likely related to the complicated method simfactory has to use to set the number of MPI ranks.
I would check if the failing test is actually runnable only on 1 MPI rank (set in test.ccl). In that case, Cactus will try to run in it in a 2 MPI rank test suite but use only 1 MPI rank. Possibly ibrun ignores Cactus' request and uses only information provided by SLURM.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Gabriele,
Great. Thank you. I will backport to the release branch.
Yours, Roland
Hello,
Reverting to srun fixes the problem. I updated the master branches for the testsuite results and simfactory.
Gabriele
On Thu, Jun 2, 2022 at 9:12 AM Gabriele Bozzola bozzola.gabriele@gmail.com wrote:
Hi Roland,
That sounds reasonable. I think I was originally using srun, but was recommended to move to ibrun. I will try with srun to see if it works, in which case I will update the simfactory entry and the testsuite results.
Gabrieel
On Thu, Jun 2, 2022 at 8:03 AM Roland Haas rhaas@illinois.edu wrote:
Hello Gabriele,
ok, I can at least partially answer this. Indeed RNS's A2 test is code to use only 1 MPI rank:
TEST rnsA2 { PROCS 1 }
and thus the most likely reason is that ibrun just pulls the number of MPI ranks from SLURM rather than from whatever simfactory tries to use.
Since ibrun is no longer documented on the SDSC page (at least I do not see it on https://urldefense.com/v3/__https://www.sdsc.edu/support/user_guides/expanse... ), maybe the easiest fix is to remove it and use the srun command they document now?
Yours, Roland
Hello Gabriele,
hmm.
/home/sbozzolo/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148: -> The environment variable CACTUS_NUM_PROCS is set to 1, but there
are 2
MPI processes. This may indicate a severe problem with the MPI startup mechanism.
IBRUN: launch command: srun -n 2 --ntasks-per-node 2
/expanse/lustre/projects/uic383/sbozzolo/ettests_2proc/SIMFACTORY/exe/cactus_sim
Looking at these, I would have expected that CACTUS_NUM_PROCS is set to 2 given that -n is 2 (being the number of MPI ranks).
The current submitscript uses ibrun though current documentation uses srun. Maybe changing to srun helps? Though the srun command does seem to have 2 MPI procs in the way you expect to.
Can you check (in the RunScript in simulations/foo/output-0000/SIMFACTORY) what CACTUS_NUM_PROCS is set to?
If this works with "regular" runs but fails with the testsuite using --testsuite then the issue is most likely related to the complicated method simfactory has to use to set the number of MPI ranks.
I would check if the failing test is actually runnable only on 1 MPI rank (set in test.ccl). In that case, Cactus will try to run in it in a 2 MPI rank test suite but use only 1 MPI rank. Possibly ibrun ignores Cactus' request and uses only information provided by SLURM.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!6AKe0V5ww0am4Al2yt... .
users@lists.einsteintoolkit.org
-
Gabriele Bozzola -
Roland Haas