<div dir="ltr"><span style="font-family:arial,sans-serif"><font size="4">Hello everyone,
</font></span><div dir="ltr" id="gmail-x_divRplyFwdMsg">
<div style="color:black">
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I am experiencing some issue in a number of GRMHD simulations of black hole binaries employing IllinoisGRMHD.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">As an example, I will write about an unequal-mass BHB configuration (with q = 2) that I'm running.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">After approximately ten orbits, the run stops with no error codes
or any other message that could help me identify the issue. The last
lines of the standard output are<br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160260e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): C2P: Lev: 9 NumPts= 569160 | Fixes: Font= 393
VL= 179 rho*= 2 | Failures: 0 InHoriz= 0 / 0 | Error: 7.124e-02,
ErrDenom: 4.838e+13 | 4.51 iters/gridpt</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160269e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Simfactory Done at date: gio 04 ago 2022 11:43:01 CEST</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I tried restarting my simulation from the latest checkpoint, but the same sudden stop occurred at the same timestep.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">At first, I thought about some problem with IGM. The last INFO is
printed by IllinoisGRMHD_driver_evaluate_MHD_rhs.C, so I put some prints
in it to identify the spot where the error occurs.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Unfortunately, I drew a blank, since the stop seems to occur just after the end of IllinoisGRMHD_driver_evaluate_MHD_rhs:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 10, Integrating to time: 3.160251e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** DEBUG END of IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Simfactory Done at date: gio 04 ago 2022 19:44:55 CEST</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I tried to restart the simulation and run it on pure MPI. It ran for a few more iterations, then stopped as well:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353565, Lev: 10, Integrating to time: 3.156831e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** DEBUG END of IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Simfactory Done at date: ven 05 ago 2022 19:00:13 CEST</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">The simulation setup is as follows:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4"> Allocated:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> Nodes: 10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> Cores per node: 48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM setting</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_NNODES : 10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_NPROCS : 20</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_NTASKS : 20</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_CPUS_ON_NODE : 48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_CPUS_PER_TASK : 24</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_TASKS_PER_NODE: 2(x10)</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> Running:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> MPI processes: 20</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> OpenMP threads per process: 24</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> MPI processes per node: 2.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> OpenMP threads per core: 1.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> OpenMP threads per node: 48</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">while the pure-MPI setup is</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4"> Allocated:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> Nodes: 10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> Cores per node: 48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM setting</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_NNODES : 10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_NPROCS : 480</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_NTASKS : 480</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_CPUS_ON_NODE : 48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_CPUS_PER_TASK : 1</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> SLURM_TASKS_PER_NODE: 48(x10)</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> Running:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> MPI processes: 480</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> OpenMP threads per process: 1</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> MPI processes per node: 48.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> OpenMP threads per core: 1.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"> OpenMP threads per node: 48</font></span></div></blockquote>
<div aria-hidden="true"><span style="font-family:arial,sans-serif"><font size="4"> </font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">I am using The Lorentz version of ET.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I've had this issue for two binary BH simulations, both
unequal-mass with q = 2. My colleague Giacomo Fedrigo experienced the
same problem running an equal-mass simulation.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I attach the q = 2 (s_UUmis_Q2) parameter file and the ET
config-info file. Also, I attach the st. error and output of my q = 2
run and of Giacomo's run (b1_UUmis_a12b_pol3_r56_gauss_9). The st.
outputs were cut for readability reasons.<br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">Can someone please help me with this?</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">Thanks in advance,</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">Federico</font></span></div></div></div></div>