<div dir="ltr"><div class="gmail_default" style="color:#000000">Dear Steven,</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">The problem is still there, which I explained in my first mail. </div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">The checkpoint issue, which you pointed out, also appears in the simulations which run successfully in our HPC while using fewer nodes (1~3). The issue with the simulation while using 9 nodes could be something else, such that it is taking too much time (24hrs +) just to set up the thorns. As far as I know, I have kept everything identical.</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font color="#666666"><br></font></div><div dir="ltr"><font color="#666666"><span class="gmail_default" style="color:rgb(0,0,0)">Regards</span></font></div><div dir="ltr"><font color="#666666"><span class="gmail_default" style="color:rgb(0,0,0)"></span>Shamim Haque</font></div><div dir="ltr"><font color="#666666">Senior Research Fellow (SRF)<br></font><div><font color="#666666">Department of Physics</font></div><div><font color="#666666">IISER Bhopal</font></div></div></div></div></div></div></div></div></div></div><br></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=f00279ea-3c03-465e-bf99-8193a73a5ee2"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 4, 2023 at 7:58 PM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu">sbrandt@cct.lsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Shamim,</p>
<p>I'm glad to see it ran on your next attempt. Did you do anything
differently?<br>
</p>
<p>--Steve<br>
</p>
<div>On 4/4/2023 3:08 AM, Shamim Haque
1910511 wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">
<div class="gmail_default">Dear Steven,</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">I assure you that I submitted the
simulation for the first time only. I used "sim
create-submit" to submit the simulation, which would not
submit the job if the same name was executed earlier.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Secondly, I found this same message
appearing in the output files from debug queue (1 node, with
GRHydro) and high memory node (3 nodes, with IllinoisGRMHD),
here the simulation ran successfully. I have attached the
output files for reference.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Regards </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Senior
Research Fellow (SRF)<br>
</font>
<div><font color="#666666">Department of
Physics</font></div>
<div><font color="#666666">IISER Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=f7761cc8-06e1-4bc7-a7be-f13a904db461"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Apr 4, 2023 at
12:35 AM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>I see this error message in your output:</p>
<p> -> [0m No HDF5 checkpoint files with basefilename
'checkpoint.chkpt' and file extension '.h5' found in
recovery directory
'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25'<br>
</p>
<p>I suspect you did a "sim submit" for a job, got a
failure, and did a second "sim submit" without purging.
That immediately triggered the error. Then, for some
reason, MPI didn't shut down cleanly and the processes
hung doing nothing until they used up the walltime.<br>
</p>
<p>--Steve<br>
</p>
<div>On 4/2/2023 5:16 AM, Shamim Haque 1910511 wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">Hello,</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">I am
trying to run BNSM using IllinoisGRMHD on HPC Kanad at
IISER Bhopal. While I have tested the parfile to be
running fine on debug queue (1 node) and high memory
queue (3 nodes), I am unable to run the simulation in
a queue with 9 nodes (144 cores). <br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">The
output file suggests that the setup of listed thorns
is not complete within 24 hours, which is the max
walltime for this queue.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Is
there a way to sort out this issue? I have attached
the parfile and outfile for reference.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Regards</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Senior
Research Fellow (SRF)<br>
</font>
<div><font color="#666666">Department
of Physics</font></div>
<div><font color="#666666">IISER
Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=9b6529d4-dc8c-422e-9133-85a74cab3e38"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div>