<div dir="ltr"><span style="font-family:arial,sans-serif"><font size="4">Hello everyone, 
</font></span><div dir="ltr" id="gmail-x_divRplyFwdMsg">
<div style="color:black">
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I am experiencing some issue in a number of GRMHD simulations of black hole binaries employing IllinoisGRMHD.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">As an example, I will write about an unequal-mass BHB configuration (with q = 2) that I&#39;m running.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">After approximately ten orbits, the run stops with no error codes 
or any other message that could help me identify the issue. The last 
lines of the standard output are<br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160260e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): C2P: Lev: 9 NumPts= 569160 | Fixes: Font= 393
 VL= 179 rho*= 2 | Failures: 0 InHoriz= 0 / 0 | Error: 7.124e-02, 
ErrDenom: 4.838e+13 | 4.51 iters/gridpt</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160269e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Simfactory Done at date: gio 04 ago 2022 11:43:01 CEST</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I tried restarting my simulation from the latest checkpoint, but the same sudden stop occurred at the same timestep.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">At first, I thought about some problem with IGM. The last INFO is 
printed by IllinoisGRMHD_driver_evaluate_MHD_rhs.C, so I put some prints
 in it to identify the spot where the error occurs.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Unfortunately, I drew a blank, since the stop seems to occur just after the end of IllinoisGRMHD_driver_evaluate_MHD_rhs:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 10, Integrating to time: 3.160251e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** DEBUG END of  IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Simfactory Done at date: gio 04 ago 2022 19:44:55 CEST</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I tried to restart the simulation and run it on pure MPI. It ran for a few more iterations, then stopped as well:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** Iter. # 353565, Lev: 10, Integrating to time: 3.156831e+03 *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">INFO (IllinoisGRMHD): ***** DEBUG END of  IllinoisGRMHD_driver_evaluate_MHD_rhs *****</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">Simfactory Done at date: ven 05 ago 2022 19:00:13 CEST</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">The simulation setup is as follows:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">   Allocated:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      Nodes:                      10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      Cores per node:             48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">   SLURM setting</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_NNODES :  10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_NPROCS :  20</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_NTASKS :  20</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_CPUS_ON_NODE  :  48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_CPUS_PER_TASK :  24</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_TASKS_PER_NODE:  2(x10)</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">   Running:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      MPI processes:              20</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      OpenMP threads per process: 24</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      MPI processes per node:     2.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      OpenMP threads per core:    1.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      OpenMP threads per node:    48</font></span></div></blockquote>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">while the pure-MPI setup is</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<blockquote style="color:rgb(102,102,102);margin-left:0.8ex;padding-left:1ex;border-color:rgb(200,200,200);border-left:3px solid rgb(200,200,200)">
<div><span style="font-family:arial,sans-serif"><font size="4">   Allocated:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      Nodes:                      10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      Cores per node:             48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">   SLURM setting</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_NNODES :  10</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_NPROCS :  480</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_NTASKS :  480</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_CPUS_ON_NODE  :  48</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_CPUS_PER_TASK :  1</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      SLURM_TASKS_PER_NODE:  48(x10)</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">   Running:</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      MPI processes:              480</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      OpenMP threads per process: 1</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      MPI processes per node:     48.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      OpenMP threads per core:    1.0</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">      OpenMP threads per node:    48</font></span></div></blockquote>
<div aria-hidden="true"><span style="font-family:arial,sans-serif"><font size="4"> </font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4">I am using The Lorentz version of ET.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I&#39;ve had this issue for two binary BH simulations, both 
unequal-mass with q = 2. My colleague Giacomo Fedrigo experienced the 
same problem running an equal-mass simulation.</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">I attach the q = 2 (s_UUmis_Q2) parameter file and the ET 
config-info file. Also, I attach the st. error and output of my q = 2 
run and of Giacomo&#39;s run (b1_UUmis_a12b_pol3_r56_gauss_9). The st. 
outputs were cut for readability reasons.<br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">Can someone please help me with this?</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">Thanks in advance,</font></span></div>
<div><span style="font-family:arial,sans-serif"><font size="4"><br aria-hidden="true"></font></span>
</div>
<div><span style="font-family:arial,sans-serif"><font size="4">Federico</font></span></div></div></div></div>