<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Ideally, all the #PBS directives should go into your
      SubmitScript. However, that's not your problem.</p>
    <p>I am wondering whether your mpirun and configured mpi match. This
      could give you the error you see.</p>
    <p>Cactus configures mpi based on which mpic++ (or mpicc) is in your
      path at the time of configuration. Did you do the same module load
      before compiling as you do for running? (module load
      /home2/mallick/ET/Cactus/openmpi-x86_64)<br>
      <br>
      Note, some builds of MPI now use the PMI interface and don't have
      an actual mpirun command. Please make sure this isn't what's going
      on here.
    </p>
    <p>Curiosity question: Is your batch queue system Torque, or are you
      using the PBS interface to slurm?</p>
    <p>--Steve<br>
    </p>
    <div class="moz-cite-prefix">On 8/12/2021 9:44 AM, Shamim Haque
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAHrQTHV2yqpj4qRxzmnY2O62pNm3=N9S0Xa9T6AfdRnZ5xQT1Q@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div class="gmail_default" style="color:#000000">For that, I
          write a separate jobscript and submit it in the queue. That
          jobscript looks like this:</div>
        <div class="gmail_default" style="color:#000000"><br>
        </div>
        <div class="gmail_default" style=""><i style=""><font
              color="#a64d79">#!/bin/bash<br>
              #PBS -N nsns2<br>
              #PBS -j oe<br>
              #PBS -V<br>
              #PBS -v TEMP=/scratch1<br>
              #PBS -o job_nsns2.out<br>
              #PBS -e job_nsns2.err<br>
              #PBS -l select=2:ncpus=16<br>
              #PBS -l walltime=24:10:00<br>
              #PBS -q p-queue<br>
              export OMP_NUM_THREADS=1<br>
              cd /home2/mallick/ET/Cactus<br>
              CPUS=`cat $PBS_NODEFILE | wc -l`<br>
              DATE=`date +%c`<br>
              echo Job started at $DATE<br>
              module load /home2/mallick/ET/Cactus/openmpi-x86_64</font></i></div>
        <div class="gmail_default" style=""><i style=""><font
              color="#a64d79"><br>
              /home2/mallick/ET/Cactus/simfactory/bin/sim whoami<br>
              <br>
            </font></i></div>
        <div class="gmail_default" style=""><i style=""><font
              color="#a64d79">time mpirun
              /home2/mallick/ET/Cactus/simfactory/bin/sim create-submit
              nsns30 --basedir=/home2/mallick/simulations --procs=32
              --ppn=16 --num-threads=1 --num-smt=1 --ppn-used=16
              --parfile
              /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par
              --walltime=24:00:00<br>
              DATE=`date +%c`<br>
              echo Job finished at $DATE</font></i></div>
        <div class="gmail_default" style=""><span
            style="color:rgb(102,102,102)"><br>
          </span></div>
        <div class="gmail_default" style=""><font color="#000000">Regards</font></div>
        <div class="gmail_default" style=""><span
            style="color:rgb(102,102,102)">Shamim Haque</span></div>
        <div>
          <div dir="ltr" class="gmail_signature"
            data-smartmail="gmail_signature">
            <div dir="ltr">
              <div dir="ltr">
                <div dir="ltr">
                  <div dir="ltr"><font color="#666666">Junior Research
                      Fellow (JRF)<br>
                    </font>
                    <div><font color="#666666">Department of Physics</font></div>
                    <div><font color="#666666">IISER Bhopal</font></div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
        <br>
      </div>
      <div hspace="streak-pt-mark" style="max-height:1px"><img alt=""
          style="width:0px;max-height:0px;overflow:hidden"
src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&amp;type=zerocontent&amp;guid=e910e754-16d4-48d2-ad25-d4cca28e9b71"
          moz-do-not-send="true"><font size="1" color="#ffffff">ᐧ</font></div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Thu, Aug 12, 2021 at 7:52
          PM Steven R. Brandt &lt;<a href="mailto:sbrandt@cct.lsu.edu"
            moz-do-not-send="true">sbrandt@cct.lsu.edu</a>&gt; wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div>
            <p>Something I'm not understanding. You want to run on 2
              nodes, but you don't seem to be using a batch queue
              system... so how does MPI know which two nodes to use?
              Does this machine have slurm installed?<br>
            </p>
            <p>--Steve<br>
            </p>
            <div>On 8/12/2021 12:28 AM, Shamim Haque wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">
                <div class="gmail_default" style="color:rgb(0,0,0)">Hi Steven,</div>
                <div class="gmail_default" style="color:rgb(0,0,0)"><br>
                </div>
                <div class="gmail_default" style="color:rgb(0,0,0)">I
                  used the generic Submitscript, no change in that. The
                  Runscript is as follows:</div>
                <div class="gmail_default" style="color:rgb(0,0,0)"><br>
                </div>
                <div class="gmail_default"><i><font color="#a64d79">echo
                      "Preparing:"<br>
                      set -x                          # Output commands<br>
                      set -e                          # Abort on errors<br>
                      cd @RUNDIR@-active<br>
                      echo "Checking:"<br>
                      pwd<br>
                      hostname<br>
                      date<br>
                      echo "Environment:"<br>
                      module load
                      /home2/mallick/ET/Cactus/openmpi-x86_64<br>
                      export CACTUS_NUM_PROCS=@NUM_PROCS@<br>
                      export CACTUS_NUM_THREADS=@NUM_THREADS@<br>
                      export GMON_OUT_PREFIX=gmon.out<br>
                      export OMP_NUM_THREADS=@NUM_THREADS@<br>
                      env | sort &gt; SIMFACTORY/ENVIRONMENT<br>
                      echo "Starting:"<br>
                      export CACTUS_STARTTIME=$(date +%s)<br>
                      mpirun -np @NUM_PROCS@ @EXECUTABLE@ -L 3 @PARFILE@<br>
                    </font></i></div>
                <div class="gmail_default"><i><font color="#a64d79">echo
                      "Stopping:"<br>
                      date<br>
                      echo "Done."<br>
                    </font></i></div>
                <div>
                  <div dir="ltr">
                    <div dir="ltr">
                      <div>
                        <div dir="ltr">
                          <div>
                            <div dir="ltr">
                              <div>
                                <div dir="ltr"><font color="#666666"><br>
                                  </font></div>
                                <div><span class="gmail_default"
                                    style="color:rgb(0,0,0)">I have
                                    attached these files here as well.
                                    All the output/error files are
                                    attached in my previous mail. </span></div>
                                <div><font color="#666666"><span
                                      class="gmail_default"
                                      style="color:rgb(0,0,0)"><br>
                                    </span></font></div>
                                <div><font color="#666666"><span
                                      class="gmail_default"
                                      style="color:rgb(0,0,0)">Regards</span></font></div>
                                <div dir="ltr"><font color="#666666">Shamim
                                    Haque</font></div>
                                <div dir="ltr"><font color="#666666">Junior
                                    Research Fellow (JRF)<br>
                                  </font>
                                  <div><font color="#666666">Department
                                      of Physics</font></div>
                                  <div><font color="#666666">IISER
                                      Bhopal</font></div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
                <br>
              </div>
              <div hspace="streak-pt-mark" style="max-height:1px"><img
                  alt="" style="width: 0px; max-height: 0px; overflow:
                  hidden;"
src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&amp;type=zerocontent&amp;guid=1bfbface-450a-422e-bbc9-02ec895c8f53"
                  moz-do-not-send="true"><font size="1" color="#ffffff">ᐧ</font></div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Thu, Aug 12, 2021
                  at 12:48 AM Steven R. Brandt &lt;<a
                    href="mailto:sbrandt@cct.lsu.edu" target="_blank"
                    moz-do-not-send="true">sbrandt@cct.lsu.edu</a>&gt;
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px 0.8ex;border-left:1px solid
                  rgb(204,204,204);padding-left:1ex">
                  <div>
                    <p>What SubmitScript and RunScript are you using?
                      Can you show us? Thanks.</p>
                    <p>--Steve<br>
                    </p>
                    <div>On 8/10/2021 2:35 AM, Shamim Haque wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)">Hello,</div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)"><br>
                        </div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)">I am trying to run
                          the BNS simulation on the cluster at IISER
                          Bhopal. Upon using 2 nodes (16x2 cores) my
                          simulation stalled at this message:</div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)"> </div>
                        <div class="gmail_default"><i><font
                              color="#a64d79">The environment variable
                              CACTUS_NUM_PROCS is set to 32, but there
                              are 1 MPI processes. This may indicate a
                              severe problem with the MPI startup
                              mechanism.</font></i></div>
                        <i><font color="#a64d79">APPLICATION TERMINATED
                            WITH THE EXIT STRING: Aborted (signal 6)</font><br>
                        </i>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)"><br>
                        </div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)">The command fed into
                          the simfactory via jobscript is as follows:</div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)"><br>
                        </div>
                        <div class="gmail_default"><font color="#a64d79"><i>time
                              mpirun
                              /home2/mallick/ET/Cactus/simfactory/bin/sim
                              create-submit nsns30
                              --basedir=/home2/mallick/simulations
                              --procs=32 --ppn=16 --num-threads=1
                              --num-smt=1 --ppn-used=16 --parfile
                              /home2/mallick/ET/Cactus/parfile/nsns_vlr_mass_diff.par
                              --walltime=24:00:00</i></font><i><b><font
                                color="#a64d79"><br>
                              </font></b></i></div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)"><br>
                        </div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)">I could not figure
                          out the issue. I am also struggling with
                          setting up the machine scripts as per the
                          cluster, so I am not sure if that is somehow
                          hampering the simulation.</div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)"><br>
                        </div>
                        <div class="gmail_default"
                          style="color:rgb(0,0,0)">Thanks in advance for
                          helping me with this issue. I have attached
                          the concerned scripts and outputs for
                          reference.</div>
                        <div class="gmail_default"><br>
                        </div>
                        <div class="gmail_default">Regards</div>
                        <div>
                          <div dir="ltr">
                            <div dir="ltr">
                              <div>
                                <div dir="ltr">
                                  <div>
                                    <div dir="ltr">
                                      <div>
                                        <div dir="ltr"><font
                                            color="#666666">Shamim Haque</font></div>
                                        <div dir="ltr"><font
                                            color="#666666">Junior
                                            Research Fellow (JRF)<br>
                                          </font>
                                          <div><font color="#666666">Department
                                              of Physics</font></div>
                                          <div><font color="#666666">IISER
                                              Bhopal</font></div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                      <div hspace="streak-pt-mark"
                        style="max-height:1px"><img alt="" style="width:
                          0px; max-height: 0px; overflow: hidden;"
src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&amp;type=zerocontent&amp;guid=5bdfad1e-6fb5-4b81-a67c-ba4f401869d6"
                          moz-do-not-send="true"><font size="1"
                          color="#ffffff">ᐧ</font></div>
                      <br>
                      <fieldset></fieldset>
                      <pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank" moz-do-not-send="true">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank" moz-do-not-send="true">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
                    </blockquote>
                  </div>
                  _______________________________________________<br>
                  Users mailing list<br>
                  <a href="mailto:Users@einsteintoolkit.org"
                    target="_blank" moz-do-not-send="true">Users@einsteintoolkit.org</a><br>
                  <a
                    href="http://lists.einsteintoolkit.org/mailman/listinfo/users"
                    rel="noreferrer" target="_blank"
                    moz-do-not-send="true">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
                </blockquote>
              </div>
            </blockquote>
          </div>
        </blockquote>
      </div>
    </blockquote>
  </body>
</html>