<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>I see this error message in your output:</p>
    <p>  -&gt; No HDF5 checkpoint files with basefilename
      'checkpoint.chkpt' and file extension '.h5' found in recovery
      directory 'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25'<br>
    </p>
    <p>I suspect you did a "sim submit" for a job, got a failure, and
      did a second "sim submit" without purging. That immediately
      triggered the error. Then, for some reason, MPI didn't shut down
      cleanly and the processes hung doing nothing until they used up
      the walltime.<br>
    </p>
    <p>--Steve<br>
    </p>
    <div class="moz-cite-prefix">On 4/2/2023 5:16 AM, Shamim Haque
      1910511 wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAHrQTHWiuD+35k4-T86G1MEuM8EFbSTcqcny6vUeZJYQOst2jQ@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div class="gmail_default" style="color:#000000">Hello,</div>
        <div class="gmail_default" style="color:#000000"><br>
        </div>
        <div class="gmail_default" style="color:#000000">I am trying to
          run BNSM using IllinoisGRMHD on HPC Kanad at IISER Bhopal.
          While I have tested the parfile to be running fine on debug
          queue (1 node) and high memory queue (3 nodes), I am unable to
          run the simulation in a queue with 9 nodes (144 cores). <br>
        </div>
        <div class="gmail_default" style="color:#000000"><br>
        </div>
        <div class="gmail_default" style="color:#000000">The output file
          suggests that the setup of listed thorns is not complete
          within 24 hours, which is the max walltime for this queue.</div>
        <div class="gmail_default" style="color:#000000"><br>
        </div>
        <div class="gmail_default" style="color:#000000">Is there a way
          to sort out this issue? I have attached the parfile and
          outfile for reference.</div>
        <div class="gmail_default" style="color:#000000"><br>
        </div>
        <div class="gmail_default" style="color:#000000">Regards</div>
        <div>
          <div dir="ltr" data-smartmail="gmail_signature">
            <div dir="ltr">
              <div>
                <div dir="ltr">
                  <div>
                    <div dir="ltr">
                      <div>
                        <div dir="ltr"><font color="#666666">Shamim
                            Haque</font></div>
                        <div dir="ltr"><font color="#666666">Senior
                            Research Fellow (SRF)<br>
                          </font>
                          <div><font color="#666666">Department of
                              Physics</font></div>
                          <div><font color="#666666">IISER Bhopal</font></div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <div hspace="streak-pt-mark" style="max-height:1px"><img alt=""
          style="width:0px;max-height:0px;overflow:hidden"
src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&amp;type=zerocontent&amp;guid=9b6529d4-dc8c-422e-9133-85a74cab3e38"
          moz-do-not-send="true"><font size="1" color="#ffffff">ᐧ</font></div>
      <br>
      <fieldset class="moz-mime-attachment-header"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a>
<a class="moz-txt-link-freetext" href="http://lists.einsteintoolkit.org/mailman/listinfo/users">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
    </blockquote>
  </body>
</html>