[ET Trac] #2960: CarpetRegrid2: failure to regrid on relative radius change - possible underlying race condition

[Jordan Nicoules]

#2960: CarpetRegrid2: failure to regrid on relative radius change - possible underlying race condition

 Reporter: Jordan Nicoules
   Status: new
Milestone: 
  Version: 
     Type: bug
 Priority: major
Component: Carpet

### Initial bug...

`CarpetRegrid2` does not properly check for a change of radius of the refinement levels (when they are steered by a `Trigger` for instance). 
Looking a bit closer, it seems that the computed `drfac` in `CarpetRegrid2/src/regrid.cc` is `nan`, and thus `do_recompose` is not set to `true`. 
In particular, it means it *is* possible to change the radii if something else triggers a regridding (for instance, changing the number of active levels).

### ... revealed a deeper issue (race condition? Memory or compiler issue?)

To figure out why `drfac` is `nan`, I added manual debugging information prints in the code. Depending on what information I ask, and in what order, I don't get consistent nor sensible results.  
*I haven't tried different compilers (I have gcc 11.4.0 locally), but this is happening even with a debug build (see attached make.config.defn file).*

More precisely, printing `rad`, `oldrad`, `rad - oldrad`, their values are fine (done through `std::cout`).
Printing (`CCTK_VINFO`) the numerator and denominator of `drfac` separately gives the right values for them, but the quotient `drfac` is wrong (it is `0`, and not `nan` anymore, even when it shouldn't).  
If I create new local variables 
```
CCTK_REAL const num = sqrt(sum(ipow(rad - oldrad, 2)));
CCTK_REAL const den = sqrt(sum(ipow(oldrad, 2)));
```
and print them, they both contain the same value. It looks like the values are as if it uses `rad=0` and whatever `oldrad` used to have before (which can be thus changed, depending on the order of the print lines).  
Finally, if I do print a lot of other info (in particular the intermediate computation steps), it seems it finally gets `drfac` right.

I had a look at `CarpetLib`'s `vect.hh` and `defs.hh` too, but I can't seem to understand what is going wrong.

### Minimal example
 
It would be helpful if someone could try to replicate the issue. 
To that effect, I'm attaching a minimal parfile (`regrid2_steer_radius.par`) adapted from `CarpetRegrid2/test/regrid2_granularity.par`, as well as a modified `regrid.cc` file with the extra `VINFO` that should allow you to quickly reproduce the situation.

I'm attaching two output files generated from these simulations, one that gets `drfac` right, and one that gets it wrong.
The difference is in the extra information I'm printing from `regrid.cc` (commented lines 840-849).

### Additional note

I stumbled upon another possible desirable correction in the thorn, although it didn't seem to affect that matter so far. Hence, I also don't know how correct it actually is.
I'm attaching the relevant `regrid.cc`, `initialise.cc` and `schedule.ccl` files in `fix__old_radius.zip`.
attachment: CarpetRegrid2_bug.zip (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2960/attachments/CarpetRegrid2_bug.zip)


--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2960/carpetregrid2-failure-to-regrid-on
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/trac/attachments/20260709/df31cb54/attachment.htm>