<div dir="ltr"><div class="gmail_default" style="color:#000000"><div class="gmail_default">Dear Steven,</div><div class="gmail_default"><br></div><div class="gmail_default">I assure you that I submitted the simulation for the first time only. I used "sim create-submit" to submit the simulation, which would not submit the job if the same name was executed earlier.</div><div class="gmail_default"><br></div><div class="gmail_default">Secondly, I found this same message appearing in the output files from debug queue (1 node, with GRHydro) and high memory node (3 nodes, with IllinoisGRMHD), here the simulation ran successfully. I have attached the output files for reference.</div><div class="gmail_default"><br></div><div class="gmail_default">Regards </div></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font color="#666666">Shamim Haque</font></div><div dir="ltr"><font color="#666666">Senior Research Fellow (SRF)<br></font><div><font color="#666666">Department of Physics</font></div><div><font color="#666666">IISER Bhopal</font></div></div></div></div></div></div></div></div></div></div><br></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=f7761cc8-06e1-4bc7-a7be-f13a904db461"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 4, 2023 at 12:35 AM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu">sbrandt@cct.lsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>I see this error message in your output:</p>
<p> -> [0m No HDF5 checkpoint files with basefilename
'checkpoint.chkpt' and file extension '.h5' found in recovery
directory 'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25'<br>
</p>
<p>I suspect you did a "sim submit" for a job, got a failure, and
did a second "sim submit" without purging. That immediately
triggered the error. Then, for some reason, MPI didn't shut down
cleanly and the processes hung doing nothing until they used up
the walltime.<br>
</p>
<p>--Steve<br>
</p>
<div>On 4/2/2023 5:16 AM, Shamim Haque
1910511 wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">Hello,</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">I am trying to
run BNSM using IllinoisGRMHD on HPC Kanad at IISER Bhopal.
While I have tested the parfile to be running fine on debug
queue (1 node) and high memory queue (3 nodes), I am unable to
run the simulation in a queue with 9 nodes (144 cores). <br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">The output file
suggests that the setup of listed thorns is not complete
within 24 hours, which is the max walltime for this queue.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Is there a way
to sort out this issue? I have attached the parfile and
outfile for reference.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Regards</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Senior
Research Fellow (SRF)<br>
</font>
<div><font color="#666666">Department of
Physics</font></div>
<div><font color="#666666">IISER Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=9b6529d4-dc8c-422e-9133-85a74cab3e38"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote></div>