[ET Trac] [Einstein Toolkit] #2182: NewRad's extrapolation methods for Gammas expect ghost zones to be valid

[Einstein Toolkit]

#2182: NewRad's extrapolation methods for Gammas expect ghost zones to be valid
---------------------------------+-----------------------------------
 Reporter:  Roland Haas          |       Type:  defect
   Status:  new                  |   Priority:  major
Milestone:  ET_2018_08           |  Component:  EinsteinToolkit thorn
  Version:  development version  |   Keywords:  ML_BSSN NewRad
---------------------------------+-----------------------------------
 The current stencil size for NewRad's extrapolation of the Xt (contracted
 Christoffel tensors) variables is 4 and the code checks for a large enough
 grid like so (extrap.cc line 61ff)
 {{{
 if (dir[d]<0) {
   assert(bmax[d] + 4 <=
          (cctkGH->cctk_bbox[2*d+1] ? imax[d] : cctkGH->cctk_lsh[d]));
 } else if (dir[d]>0) {
   assert(bmin[d] - 4 >= (cctkGH->cctk_bbox[2*d] ? imin[d] : 0));
 }
 }}}
 which checks (eg for {{{dir[d] > 0}}} ie an upper boundary) that, unless
 the *lower* boundary is a grid boundary, that there are 4 points between
 bmin and the beginning of the patch (0).

 This therefore assumes that ghost points (points 0...cctk_nghostzones) are
 valid since they will be used if the grid is small enough (less than
 4+2*cctk_nghostzones).

 Currently this is not ensured by McLachlan and only points in the interior
 are valid.

 This causes the attached parfile to produce poison in the output if run
 with 2 MPI ranks.

 A fix would be to add a {{{SYNC: ML_Gamma}}} to ML_BSSN's
 ML_BSSN_InitialADMBase2Interior routine or to make the check in NewRad
 stricter, requiring that there are at least 4 points in the interior
 (which would also make the check simpler).

 My feeling would be that this also explains NaNs that people have seen
 when running a TOV simulation with many MPI ranks.

 I attach a sample parfile as well as sample output.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/2182>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit