#2182: NewRad's extrapolation methods for Gammas expect ghost zones to be valid ---------------------------------+----------------------------------- Reporter: Roland Haas | Type: defect Status: new | Priority: major Milestone: ET_2018_08 | Component: EinsteinToolkit thorn Version: development version | Keywords: ML_BSSN NewRad ---------------------------------+----------------------------------- The current stencil size for NewRad's extrapolation of the Xt (contracted Christoffel tensors) variables is 4 and the code checks for a large enough grid like so (extrap.cc line 61ff) {{{ if (dir[d]<0) { assert(bmax[d] + 4 <= (cctkGH->cctk_bbox[2*d+1] ? imax[d] : cctkGH->cctk_lsh[d])); } else if (dir[d]>0) { assert(bmin[d] - 4 >= (cctkGH->cctk_bbox[2*d] ? imin[d] : 0)); } }}} which checks (eg for {{{dir[d] > 0}}} ie an upper boundary) that, unless the *lower* boundary is a grid boundary, that there are 4 points between bmin and the beginning of the patch (0).
This therefore assumes that ghost points (points 0...cctk_nghostzones) are valid since they will be used if the grid is small enough (less than 4+2*cctk_nghostzones).
Currently this is not ensured by McLachlan and only points in the interior are valid.
This causes the attached parfile to produce poison in the output if run with 2 MPI ranks.
A fix would be to add a {{{SYNC: ML_Gamma}}} to ML_BSSN's ML_BSSN_InitialADMBase2Interior routine or to make the check in NewRad stricter, requiring that there are at least 4 points in the interior (which would also make the check simpler).
My feeling would be that this also explains NaNs that people have seen when running a TOV simulation with many MPI ranks.
I attach a sample parfile as well as sample output.