<div dir="ltr"><div class="gmail_default" style="color:#000000">For that, I write a separate jobscript and submit it in the queue. That jobscript looks like this:</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style=""><i style=""><font color="#a64d79">#!/bin/bash<br>#PBS -N nsns2<br>#PBS -j oe<br>#PBS -V<br>#PBS -v TEMP=/scratch1<br>#PBS -o job_nsns2.out<br>#PBS -e job_nsns2.err<br>#PBS -l select=2:ncpus=16<br>#PBS -l walltime=24:10:00<br>#PBS -q p-queue<br>export OMP_NUM_THREADS=1<br>cd /home2/mallick/ET/Cactus<br>CPUS=`cat $PBS_NODEFILE | wc -l`<br>DATE=`date +%c`<br>echo Job started at $DATE<br>module load /home2/mallick/ET/Cactus/openmpi-x86_64</font></i></div><div class="gmail_default" style=""><i style=""><font color="#a64d79"><br>/home2/mallick/ET/Cactus/simfactory/bin/sim whoami<br><br></font></i></div><div class="gmail_default" style=""><i style=""><font color="#a64d79">time mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit nsns30 --basedir=/home2/mallick/simulations --procs=32 --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16 --parfile /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par --walltime=24:00:00<br>DATE=`date +%c`<br>echo Job finished at $DATE</font></i></div><div class="gmail_default" style=""><span style="color:rgb(102,102,102)"><br></span></div><div class="gmail_default" style=""><font color="#000000">Regards</font></div><div class="gmail_default" style=""><span style="color:rgb(102,102,102)">Shamim Haque</span></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font color="#666666">Junior Research Fellow (JRF)<br></font><div><font color="#666666">Department of Physics</font></div><div><font color="#666666">IISER Bhopal</font></div></div></div></div></div></div></div><br></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=e910e754-16d4-48d2-ad25-d4cca28e9b71"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 12, 2021 at 7:52 PM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu">sbrandt@cct.lsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Something I'm not understanding. You want to run on 2 nodes, but
you don't seem to be using a batch queue system... so how does MPI
know which two nodes to use? Does this machine have slurm
installed?<br>
</p>
<p>--Steve<br>
</p>
<div>On 8/12/2021 12:28 AM, Shamim Haque
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">Hi Steven,</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">I used the
generic Submitscript, no change in that. The Runscript is as
follows:</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default"><i><font color="#a64d79">echo "Preparing:"<br>
set -x # Output commands<br>
set -e # Abort on errors<br>
cd @RUNDIR@-active<br>
echo "Checking:"<br>
pwd<br>
hostname<br>
date<br>
echo "Environment:"<br>
module load /home2/mallick/ET/Cactus/openmpi-x86_64<br>
export CACTUS_NUM_PROCS=@NUM_PROCS@<br>
export CACTUS_NUM_THREADS=@NUM_THREADS@<br>
export GMON_OUT_PREFIX=gmon.out<br>
export OMP_NUM_THREADS=@NUM_THREADS@<br>
env | sort > SIMFACTORY/ENVIRONMENT<br>
echo "Starting:"<br>
export CACTUS_STARTTIME=$(date +%s)<br>
mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@<br>
</font></i></div>
<div class="gmail_default"><i><font color="#a64d79">echo
"Stopping:"<br>
date<br>
echo "Done."<br>
</font></i></div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666"><br>
</font></div>
<div><span class="gmail_default" style="color:rgb(0,0,0)">I have attached
these files here as well. All the
output/error files are attached in my
previous mail. </span></div>
<div><font color="#666666"><span class="gmail_default" style="color:rgb(0,0,0)"><br>
</span></font></div>
<div><font color="#666666"><span class="gmail_default" style="color:rgb(0,0,0)">Regards</span></font></div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Junior
Research Fellow (JRF)<br>
</font>
<div><font color="#666666">Department of
Physics</font></div>
<div><font color="#666666">IISER Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=1bfbface-450a-422e-bbc9-02ec895c8f53"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 12, 2021 at 12:48
AM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>What SubmitScript and RunScript are you using? Can you
show us? Thanks.</p>
<p>--Steve<br>
</p>
<div>On 8/10/2021 2:35 AM, Shamim Haque wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">Hello,</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">I am
trying to run the BNS simulation on the cluster at
IISER Bhopal. Upon using 2 nodes (16x2 cores) my
simulation stalled at this message:</div>
<div class="gmail_default" style="color:rgb(0,0,0)"> </div>
<div class="gmail_default"><i><font color="#a64d79">The
environment variable CACTUS_NUM_PROCS is set to
32, but there are 1 MPI processes. This may
indicate a severe problem with the MPI startup
mechanism.</font></i></div>
<i><font color="#a64d79">APPLICATION TERMINATED WITH THE
EXIT STRING: Aborted (signal 6)</font><br>
</i>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">The
command fed into the simfactory via jobscript is as
follows:</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default"><font color="#a64d79"><i>time
mpirun /home2/mallick/ET/Cactus/simfactory/bin/sim
create-submit nsns30
--basedir=/home2/mallick/simulations --procs=32
--ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16
--parfile
/home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par
--walltime=24:00:00</i></font><i><b><font color="#a64d79"><br>
</font></b></i></div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">I
could not figure out the issue. I am also struggling
with setting up the machine scripts as per the
cluster, so I am not sure if that is somehow hampering
the simulation.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Thanks
in advance for helping me with this issue. I have
attached the concerned scripts and outputs for
reference.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Regards</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Junior
Research Fellow (JRF)<br>
</font>
<div><font color="#666666">Department
of Physics</font></div>
<div><font color="#666666">IISER
Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=5bdfad1e-6fb5-4b81-a67c-ba4f401869d6"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div>