For that, I write a separate jobscript and submit it in the queue. That jobscript looks like this:
*#!/bin/bash#PBS -N nsns2#PBS -j oe#PBS -V#PBS -v TEMP=/scratch1#PBS -o job_nsns2.out#PBS -e job_nsns2.err#PBS -l select=2:ncpus=16#PBS -l walltime=24:10:00#PBS -q p-queueexport OMP_NUM_THREADS=1cd /home2/mallick/ET/CactusCPUS=`cat $PBS_NODEFILE | wc -l`DATE=`date +%c`echo Job started at $DATEmodule load /home2/mallick/ET/Cactus/openmpi-x86_64*
*/home2/mallick/ET/Cactus/simfactory/bin/sim whoami*
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00DATE=`date +%c`echo Job finished at $DATE*
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 7:52 PM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
Something I'm not understanding. You want to run on 2 nodes, but you don't seem to be using a batch queue system... so how does MPI know which two nodes to use? Does this machine have slurm installed?
--Steve On 8/12/2021 12:28 AM, Shamim Haque wrote:
Hi Steven,
I used the generic Submitscript, no change in that. The Runscript is as follows:
*echo "Preparing:" set -x # Output commands set -e # Abort on errors cd @RUNDIR@-active echo "Checking:" pwd hostname date echo "Environment:" module load /home2/mallick/ET/Cactus/openmpi-x86_64 export CACTUS_NUM_PROCS=@NUM_PROCS@ export CACTUS_NUM_THREADS=@NUM_THREADS@ export GMON_OUT_PREFIX=gmon.out export OMP_NUM_THREADS=@NUM_THREADS@ env | sort > SIMFACTORY/ENVIRONMENT echo "Starting:" export CACTUS_STARTTIME=$(date +%s) mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@ *
*echo "Stopping:" date echo "Done." *
I have attached these files here as well. All the output/error files are attached in my previous mail.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Aug 12, 2021 at 12:48 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:
What SubmitScript and RunScript are you using? Can you show us? Thanks.
--Steve On 8/10/2021 2:35 AM, Shamim Haque wrote:
Hello,
I am trying to run the BNS simulation on the cluster at IISER Bhopal. Upon using 2 nodes (16x2 cores) my simulation stalled at this message:
*The environment variable CACTUS_NUM_PROCS is set to 32, but there are 1 MPI processes. This may indicate a severe problem with the MPI startup mechanism.*
*APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) *
The command fed into the simfactory via jobscript is as follows:
*time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00*
I could not figure out the issue. I am also struggling with setting up the machine scripts as per the cluster, so I am not sure if that is somehow hampering the simulation.
Thanks in advance for helping me with this issue. I have attached the concerned scripts and outputs for reference.
Regards Shamim Haque Junior Research Fellow (JRF) Department of Physics IISER Bhopal ᐧ
Users mailing listUsers@einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users