#2960: CarpetRegrid2: failure to regrid on relative radius change - possible underlying race condition

Reporter: Jordan Nicoules Status: new Milestone: Version: Type: bug Priority: major Component: Carpet

### Initial bug...

`CarpetRegrid2` does not properly check for a change of radius of the refinement levels (when they are steered by a `Trigger` for instance). Looking a bit closer, it seems that the computed `drfac` in `CarpetRegrid2/src/regrid.cc` is `nan`, and thus `do_recompose` is not set to `true`. In particular, it means it *is* possible to change the radii if something else triggers a regridding (for instance, changing the number of active levels).

### ... revealed a deeper issue (race condition? Memory or compiler issue?)

To figure out why `drfac` is `nan`, I added manual debugging information prints in the code. Depending on what information I ask, and in what order, I don't get consistent nor sensible results. *I haven't tried different compilers (I have gcc 11.4.0 locally), but this is happening even with a debug build (see attached make.config.defn file).*

More precisely, printing `rad`, `oldrad`, `rad - oldrad`, their values are fine (done through `std::cout`). Printing (`CCTK_VINFO`) the numerator and denominator of `drfac` separately gives the right values for them, but the quotient `drfac` is wrong (it is `0`, and not `nan` anymore, even when it shouldn't). If I create new local variables ``` CCTK_REAL const num = sqrt(sum(ipow(rad - oldrad, 2))); CCTK_REAL const den = sqrt(sum(ipow(oldrad, 2))); ``` and print them, they both contain the same value. It looks like the values are as if it uses `rad=0` and whatever `oldrad` used to have before (which can be thus changed, depending on the order of the print lines). Finally, if I do print a lot of other info (in particular the intermediate computation steps), it seems it finally gets `drfac` right.

I had a look at `CarpetLib`'s `vect.hh` and `defs.hh` too, but I can't seem to understand what is going wrong.

### Minimal example

It would be helpful if someone could try to replicate the issue. To that effect, I'm attaching a minimal parfile (`regrid2_steer_radius.par`) adapted from `CarpetRegrid2/test/regrid2_granularity.par`, as well as a modified `regrid.cc` file with the extra `VINFO` that should allow you to quickly reproduce the situation.

I'm attaching two output files generated from these simulations, one that gets `drfac` right, and one that gets it wrong. The difference is in the extra information I'm printing from `regrid.cc` (commented lines 840-849).

### Additional note

I stumbled upon another possible desirable correction in the thorn, although it didn't seem to affect that matter so far. Hence, I also don't know how correct it actually is. I'm attaching the relevant `regrid.cc`, `initialise.cc` and `schedule.ccl` files in `fix__old_radius.zip`. attachment: CarpetRegrid2_bug.zip (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/29...)

-- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2960/carpetregrid2-fail...