[Users] Issue with black hole binary evolution employing IGM

[Federico Cattorini]

Hello everyone,

I am experiencing some issue in a number of GRMHD simulations of black hole
binaries employing IllinoisGRMHD.

As an example, I will write about an unequal-mass BHB configuration (with q
= 2) that I'm running.

After approximately ten orbits, the run stops with no error codes or any
other message that could help me identify the issue. The last lines of the
standard output are

INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time:
3.160260e+03 *****
INFO (IllinoisGRMHD): C2P: Lev: 9 NumPts= 569160 | Fixes: Font= 393 VL= 179
rho*= 2 | Failures: 0 InHoriz= 0 / 0 | Error: 7.124e-02, ErrDenom:
4.838e+13 | 4.51 iters/gridpt
INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time:
3.160269e+03 *****
Simfactory Done at date: gio 04 ago 2022 11:43:01 CEST



I tried restarting my simulation from the latest checkpoint, but the same
sudden stop occurred at the same timestep.

At first, I thought about some problem with IGM. The last INFO is printed
by IllinoisGRMHD_driver_evaluate_MHD_rhs.C, so I put some prints in it to
identify the spot where the error occurs.
Unfortunately, I drew a blank, since the stop seems to occur just after the
end of IllinoisGRMHD_driver_evaluate_MHD_rhs:

INFO (IllinoisGRMHD): ***** line 52: entering
IllinoisGRMHD_driver_evaluate_MHD_rhs *****
INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 10, Integrating to time:
3.160251e+03 *****
INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs
*****
INFO (IllinoisGRMHD): ***** line 204: just before
reconstruct_set_of_prims_PPM *****
INFO (IllinoisGRMHD): ***** DEBUG END of
 IllinoisGRMHD_driver_evaluate_MHD_rhs *****
Simfactory Done at date: gio 04 ago 2022 19:44:55 CEST


I tried to restart the simulation and run it on pure MPI. It ran for a few
more iterations, then stopped as well:

INFO (IllinoisGRMHD): ***** line 52: entering
IllinoisGRMHD_driver_evaluate_MHD_rhs *****
INFO (IllinoisGRMHD): ***** Iter. # 353565, Lev: 10, Integrating to time:
3.156831e+03 *****
INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs
*****
INFO (IllinoisGRMHD): ***** line 204: just before
reconstruct_set_of_prims_PPM *****
INFO (IllinoisGRMHD): ***** DEBUG END of
 IllinoisGRMHD_driver_evaluate_MHD_rhs *****
Simfactory Done at date: ven 05 ago 2022 19:00:13 CEST


The simulation setup is as follows:

   Allocated:
      Nodes:                      10
      Cores per node:             48
   SLURM setting
      SLURM_NNODES :  10
      SLURM_NPROCS :  20
      SLURM_NTASKS :  20
      SLURM_CPUS_ON_NODE  :  48
      SLURM_CPUS_PER_TASK :  24
      SLURM_TASKS_PER_NODE:  2(x10)
   Running:
      MPI processes:              20
      OpenMP threads per process: 24
      MPI processes per node:     2.0
      OpenMP threads per core:    1.0
      OpenMP threads per node:    48


while the pure-MPI setup is

   Allocated:
      Nodes:                      10
      Cores per node:             48
   SLURM setting
      SLURM_NNODES :  10
      SLURM_NPROCS :  480
      SLURM_NTASKS :  480
      SLURM_CPUS_ON_NODE  :  48
      SLURM_CPUS_PER_TASK :  1
      SLURM_TASKS_PER_NODE:  48(x10)
   Running:
      MPI processes:              480
      OpenMP threads per process: 1
      MPI processes per node:     48.0
      OpenMP threads per core:    1.0
      OpenMP threads per node:    48


I am using The Lorentz version of ET.

I've had this issue for two binary BH simulations, both unequal-mass with q
= 2. My colleague Giacomo Fedrigo experienced the same problem running an
equal-mass simulation.

I attach the q = 2 (s_UUmis_Q2) parameter file and the ET config-info file.
Also, I attach the st. error and output of my q = 2 run and of Giacomo's
run (b1_UUmis_a12b_pol3_r56_gauss_9). The st. outputs were cut for
readability reasons.

Can someone please help me with this?

Thanks in advance,

Federico
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config-info
Type: application/octet-stream
Size: 2881 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0006.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s_DDmis_q2.par
Type: application/octet-stream
Size: 21177 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0007.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s_UUmis_Q2.out
Type: application/octet-stream
Size: 161816 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0008.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: b1_UUmis_a12b_pol3_r56_gauss_9.out
Type: application/octet-stream
Size: 167833 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0009.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s_UUmis_Q2.err
Type: application/octet-stream
Size: 141898 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0010.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: b1_UUmis_a12b_pol3_r56_gauss_9.err
Type: application/octet-stream
Size: 328242 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20220914/58e369ea/attachment-0011.obj