<div dir="ltr"><div class="gmail_default" style="color:#000000">Dear Steve,</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">Thank you for your reply. I tried the same simulation with a finer grid, and the simulation started working fine, even though very slow (looks like due to slow inter-node communication), but it did work out. I could see a few iterations towards the final couple of hours from the wall time. </div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">Turns out, a simulation with GRhydro, in such cases (where the grid needs to be finer), would end with an error saying, "<i>the grid structure inconsistent. Impossible to continue</i>". On the other hand, a simulation with IllinoisGRMHD stops abruptly during the thorn setup (somewhere around the SpaceMask and AHFinderDirect setup).</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">Later I tried to see if I can pace up the simulation, but looks like the inter-node communication is very slow in the HPC, which may be an inherent problem with the HPC since it is a very old one.</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">Regards</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font color="#666666">Shamim Haque</font></div><div dir="ltr"><font color="#666666">Senior Research Fellow (SRF)<br></font><div><font color="#666666">Department of Physics</font></div><div><font color="#666666">IISER Bhopal</font></div></div></div></div></div></div></div></div></div></div><br></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=9e9988bb-8e9d-48d7-a3af-eccaf5ed5784"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 23, 2023 at 10:08 PM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu">sbrandt@cct.lsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Sorry that no one has replied to you in a while. Are you still
experiencing this difficulty?</p>
<p>--Steve<br>
</p>
<div>On 4/4/2023 3:08 AM, Shamim Haque
1910511 wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">
<div class="gmail_default">Dear Steven,</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">I assure you that I submitted the
simulation for the first time only. I used "sim
create-submit" to submit the simulation, which would not
submit the job if the same name was executed earlier.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Secondly, I found this same message
appearing in the output files from debug queue (1 node, with
GRHydro) and high memory node (3 nodes, with IllinoisGRMHD),
here the simulation ran successfully. I have attached the
output files for reference.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Regards </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Senior
Research Fellow (SRF)<br>
</font>
<div><font color="#666666">Department of
Physics</font></div>
<div><font color="#666666">IISER Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=f7761cc8-06e1-4bc7-a7be-f13a904db461"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Apr 4, 2023 at
12:35 AM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>I see this error message in your output:</p>
<p> -> [0m No HDF5 checkpoint files with basefilename
'checkpoint.chkpt' and file extension '.h5' found in
recovery directory
'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25'<br>
</p>
<p>I suspect you did a "sim submit" for a job, got a
failure, and did a second "sim submit" without purging.
That immediately triggered the error. Then, for some
reason, MPI didn't shut down cleanly and the processes
hung doing nothing until they used up the walltime.<br>
</p>
<p>--Steve<br>
</p>
<div>On 4/2/2023 5:16 AM, Shamim Haque 1910511 wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(0,0,0)">Hello,</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">I am
trying to run BNSM using IllinoisGRMHD on HPC Kanad at
IISER Bhopal. While I have tested the parfile to be
running fine on debug queue (1 node) and high memory
queue (3 nodes), I am unable to run the simulation in
a queue with 9 nodes (144 cores). <br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">The
output file suggests that the setup of listed thorns
is not complete within 24 hours, which is the max
walltime for this queue.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Is
there a way to sort out this issue? I have attached
the parfile and outfile for reference.</div>
<div class="gmail_default" style="color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default" style="color:rgb(0,0,0)">Regards</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><font color="#666666">Shamim
Haque</font></div>
<div dir="ltr"><font color="#666666">Senior
Research Fellow (SRF)<br>
</font>
<div><font color="#666666">Department
of Physics</font></div>
<div><font color="#666666">IISER
Bhopal</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width: 0px; max-height: 0px; overflow: hidden;" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=9b6529d4-dc8c-422e-9133-85a74cab3e38"><font size="1" color="#ffffff">ᐧ</font></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div>