Hello,
I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message:
*The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism.*
*APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)*
The command fed into the simfactory via jobscript is as follows:
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00*
I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation.
Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ
What SubmitScript and RunScript are you using? Can you show us? Thanks.
--Steve
On 8/10/2021 2:35 AM, Shamim Haque wrote:
Hello,
I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message: /The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism./ /APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) /
The command fed into the simfactory via jobscript is as follows:
/time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00//* */
I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation.
Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Steven,
I used the generic Submitscript, no change in that. The Runscript is as follows:
*echo "Preparing:"set -x # Output commandsset -e # Abort on errorscd @RUNDIR@-activeecho "Checking:"pwdhostnamedateecho "Environment:"module load /home2/mallick/ET/Cactus/openmpi-x86_64export CACTUS_NUM_PROCS=@NUM_PROCS@export CACTUS_NUM_THREADS=@NUM_THREADS@export GMON_OUT_PREFIX=gmon.outexport OMP_NUM_THREADS=@NUM_THREADS@env | sort > SIMFACTORY/ENVIRONMENTecho "Starting:"export CACTUS_STARTTIME=$(date +%s)mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@*
*echo "Stopping:"dateecho "Done."*
I have attached these files here as well. All the output/error files are attached in my previous mail.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 12:48 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
What SubmitScript and RunScript are you using? Can you show us? Thanks.
--Steve On 8/10/2021 2:35 AM, Shamim Haque wrote:
Hello,
I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message:
*The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism.*
*APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) *
The command fed into the simfactory via jobscript is as follows:
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00*
I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation.
Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Something I'm not understanding. You want to run on 2 nodes, but you don't seem to be using a batch queue system... so how does MPI know which two nodes to use? Does this machine have slurm installed?
--Steve
On 8/12/2021 12:28 AM, Shamim Haque wrote:
Hi Steven,
I used the generic Submitscript, no change in that. The Runscript is as follows:
/echo "Preparing:" set -x # Output commands set -e # Abort on errors cd @RUNDIR@-active echo "Checking:" pwd hostname date echo "Environment:" module load /home2/mallick/ET/Cactus/openmpi-x86_64 export CACTUS_NUM_PROCS=@NUM_PROCS@ export CACTUS_NUM_THREADS=@NUM_THREADS@ export GMON_OUT_PREFIX=gmon.out export OMP_NUM_THREADS=@NUM_THREADS@ env | sort > SIMFACTORY/ENVIRONMENT echo "Starting:" export CACTUS_STARTTIME=$(date +%s) mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@ / /echo "Stopping:" date echo "Done." /
I have attached these files here as well. All the output/error files are attached in my previous mail.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 12:48 AM Steven R. Brandt <sbrandt@cct.lsu.edu mailto:sbrandt@cct.lsu.edu> wrote:
What SubmitScript and RunScript are you using? Can you show us? Thanks. --Steve On 8/10/2021 2:35 AM, Shamim Haque wrote:Hello, I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message: /The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism./ /APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) / The command fed into the simfactory via jobscript is as follows: /time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00//* */ I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation. Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference. Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users <http://lists.einsteintoolkit.org/mailman/listinfo/users>_______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users <http://lists.einsteintoolkit.org/mailman/listinfo/users>
For that, I write a separate jobscript and submit it in the queue. That jobscript looks like this:
*#!/bin/bash#PBS -N nsns2#PBS -j oe#PBS -V#PBS -v TEMP=/scratch1#PBS -o job_nsns2.out#PBS -e job_nsns2.err#PBS -l select=2:ncpus=16#PBS -l walltime=24:10:00#PBS -q p-queueexport OMP_NUM_THREADS=1cd /home2/mallick/ET/CactusCPUS=`cat $PBS_NODEFILE | wc -l`DATE=`date +%c`echo Job started at $DATEmodule load /home2/mallick/ET/Cactus/openmpi-x86_64*
*/home2/mallick/ET/Cactus/simfactory/bin/sim whoami*
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00DATE=`date +%c`echo Job finished at $DATE*
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 7:52 PM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
Something I'm not understanding. You want to run on 2 nodes, but you don't seem to be using a batch queue system... so how does MPI know which two nodes to use? Does this machine have slurm installed?
--Steve On 8/12/2021 12:28 AM, Shamim Haque wrote:
Hi Steven,
I used the generic Submitscript, no change in that. The Runscript is as follows:
*echo "Preparing:" set -x # Output commands set -e # Abort on errors cd @RUNDIR@-active echo "Checking:" pwd hostname date echo "Environment:" module load /home2/mallick/ET/Cactus/openmpi-x86_64 export CACTUS_NUM_PROCS=@NUM_PROCS@ export CACTUS_NUM_THREADS=@NUM_THREADS@ export GMON_OUT_PREFIX=gmon.out export OMP_NUM_THREADS=@NUM_THREADS@ env | sort > SIMFACTORY/ENVIRONMENT echo "Starting:" export CACTUS_STARTTIME=$(date +%s) mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@ *
*echo "Stopping:" date echo "Done." *
I have attached these files here as well. All the output/error files are attached in my previous mail.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 12:48 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
What SubmitScript and RunScript are you using? Can you show us? Thanks.
--Steve On 8/10/2021 2:35 AM, Shamim Haque wrote:
Hello,
I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message:
*The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism.*
*APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) *
The command fed into the simfactory via jobscript is as follows:
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00*
I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation.
Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello all,
these error messages are often caused by using a different MPI stack when compiling then when running.
You would have to make sure that eg the same modules are loaded.
You could check that eg the location of the MPI library (from ldd cactus_sim) and mpirun (from which mpirun) are compatible (eg in the same subdirectory tree).
Yours, Roland
For that, I write a separate jobscript and submit it in the queue. That jobscript looks like this:
*#!/bin/bash#PBS -N nsns2#PBS -j oe#PBS -V#PBS -v TEMP=/scratch1#PBS -o job_nsns2.out#PBS -e job_nsns2.err#PBS -l select=2:ncpus=16#PBS -l walltime=24:10:00#PBS -q p-queueexport OMP_NUM_THREADS=1cd /home2/mallick/ET/CactusCPUS=`cat $PBS_NODEFILE | wc -l`DATE=`date +%c`echo Job started at $DATEmodule load /home2/mallick/ET/Cactus/openmpi-x86_64*
*/home2/mallick/ET/Cactus/simfactory/bin/sim whoami*
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00DATE=`date +%c`echo Job finished at $DATE*
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 7:52 PM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
Something I'm not understanding. You want to run on 2 nodes, but you don't seem to be using a batch queue system... so how does MPI know which two nodes to use? Does this machine have slurm installed?
--Steve On 8/12/2021 12:28 AM, Shamim Haque wrote:
Hi Steven,
I used the generic Submitscript, no change in that. The Runscript is as follows:
*echo "Preparing:" set -x # Output commands set -e # Abort on errors cd @RUNDIR@-active echo "Checking:" pwd hostname date echo "Environment:" module load /home2/mallick/ET/Cactus/openmpi-x86_64 export CACTUS_NUM_PROCS=@NUM_PROCS@ export CACTUS_NUM_THREADS=@NUM_THREADS@ export GMON_OUT_PREFIX=gmon.out export OMP_NUM_THREADS=@NUM_THREADS@ env | sort > SIMFACTORY/ENVIRONMENT echo "Starting:" export CACTUS_STARTTIME=$(date +%s) mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@ *
*echo "Stopping:" date echo "Done." *
I have attached these files here as well. All the output/error files are attached in my previous mail.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 12:48 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
What SubmitScript and RunScript are you using? Can you show us? Thanks.
--Steve On 8/10/2021 2:35 AM, Shamim Haque wrote:
Hello,
I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message:
*The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism.*
*APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) *
The command fed into the simfactory via jobscript is as follows:
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00*
I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation.
Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
Ideally, all the #PBS directives should go into your SubmitScript. However, that's not your problem.
I am wondering whether your mpirun and configured mpi match. This could give you the error you see.
Cactus configures mpi based on which mpic++ (or mpicc) is in your path at the time of configuration. Did you do the same module load before compiling as you do for running? (module load /home2/mallick/ET/Cactus/openmpi-x86_64)
Note, some builds of MPI now use the PMI interface and don't have an actual mpirun command. Please make sure this isn't what's going on here.
Curiosity question: Is your batch queue system Torque, or are you using the PBS interface to slurm?
--Steve
On 8/12/2021 9:44 AM, Shamim Haque wrote:
For that, I write a separate jobscript and submit it in the queue. That jobscript looks like this:
/#!/bin/bash #PBS -N nsns2 #PBS -j oe #PBS -V #PBS -v TEMP=/scratch1 #PBS -o job_nsns2.out #PBS -e job_nsns2.err #PBS -l select=2:ncpus=16 #PBS -l walltime=24:10:00 #PBS -q p-queue export OMP_NUM_THREADS=1 cd /home2/mallick/ET/Cactus CPUS=`cat $PBS_NODEFILE | wc -l` DATE=`date +%c` echo Job started at $DATE module load /home2/mallick/ET/Cactus/openmpi-x86_64/ / /home2/mallick/ET/Cactus/simfactory/bin/sim whoami
/ /time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00 DATE=`date +%c` echo Job finished at $DATE/
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 7:52 PM Steven R. Brandt <sbrandt@cct.lsu.edu mailto:sbrandt@cct.lsu.edu> wrote:
Something I'm not understanding. You want to run on 2 nodes, but you don't seem to be using a batch queue system... so how does MPI know which two nodes to use? Does this machine have slurm installed? --Steve On 8/12/2021 12:28 AM, Shamim Haque wrote:Hi Steven, I used the generic Submitscript, no change in that. The Runscript is as follows: /echo "Preparing:" set -x # Output commands set -e # Abort on errors cd @RUNDIR@-active echo "Checking:" pwd hostname date echo "Environment:" module load /home2/mallick/ET/Cactus/openmpi-x86_64 export CACTUS_NUM_PROCS=@NUM_PROCS@ export CACTUS_NUM_THREADS=@NUM_THREADS@ export GMON_OUT_PREFIX=gmon.out export OMP_NUM_THREADS=@NUM_THREADS@ env | sort > SIMFACTORY/ENVIRONMENT echo "Starting:" export CACTUS_STARTTIME=$(date +%s) mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@ / /echo "Stopping:" date echo "Done." / I have attached these files here as well. All the output/error files are attached in my previous mail. Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ On Thu, Aug 12, 2021 at 12:48 AM Steven R. Brandt <sbrandt@cct.lsu.edu <mailto:sbrandt@cct.lsu.edu>> wrote: What SubmitScript and RunScript are you using? Can you show us? Thanks. --Steve On 8/10/2021 2:35 AM, Shamim Haque wrote:Hello, I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message: /The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism./ /APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) / The command fed into the simfactory via jobscript is as follows: /time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00//* */ I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation. Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference. Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users <http://lists.einsteintoolkit.org/mailman/listinfo/users>_______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users <http://lists.einsteintoolkit.org/mailman/listinfo/users>
users@lists.einsteintoolkit.org
-
Roland Haas -
Shamim Haque -
Steven R. Brandt