Changes (by Miren Radia):

Hello,

I’ve been trying to run a BH binary simulation with CarpetX on tursa (Nvidia A100 GPUs) but I seem to be running into problems with the code crashing/assertions failing that I can’t resolve. I have been using Liwei Ji’s development branches in order to use subcycling (following the guidance described here) so it’s possible my problems are related to this.

Please find attached the following:

• My thornlist: carpetx-subcycling.th
• My parameter file: test-bbh.par (this is a very low resolution/small configuration just to reproduce the crash. I was originally trying something bigger)
• My OptionsList: tursa.cfg
• The full output from running the code through cuda-gdb including a backtrace at the failed assertion and printing out some variables that seem to be leading to the failed assertion: test-bbh-debug.out

Here’s the end of the output printed from the simulation (I have increased the verbosity):

INFO (CarpetX): Starting evolution...
INFO (CarpetX): Regridding...
INFO (CarpetX): Setting max_grid_size values for all levels before regridding
INFO (CarpetX): ErrorEst patch 0 level 0
INFO (CarpetX): ErrorEst patch 0 level 0 done. Set/clear/total=288/3808/4096=7%/93%/100%
INFO (CarpetX):   old levels 2, new levels 2
INFO (CarpetX):   level 0: 1 boxes, 4096 cells (100%)
INFO (CarpetX):   level 1: 1 boxes, 8192 cells (25%, 25%)
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL
INFO (CarpetX): CallFunction iteration 1 C
--

Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2890/carpetx-crash-failed-assertion-in-z4c_rhs