#2030: Multi-block boundaries leave uninitialized boundary points ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by hinder):
When syncing ghost zones (for interprocess communication), each ghost point can be updated in precisely one way, which is to set its value to the value held by the process which "owns" the real point. i.e., we have only one "boundary condition" which can be applied to that point. A "standard" interpatch boundary point is the same; in the usual case, there is a single component on a single process which owns the point, and the boundary point is filled by interpolation from the neighbouring points in that component. Similarly, a "standard" outer boundary point can be filled by applying the physical boundary condition. However, a problem arises when a point is BOTH an outer boundary point and an interpatch boundary point. This would happen for a point at r = r_max and phi = phi_max.
I think this problem only happens during initialisation, not evolution. The steps are these:
1. CCTK_INITIAL/ADMBase_InitialData: TwoPunctures initialises the ADMBase metric variables pointwise at every point on the grid. 2. CCTK_INITIAL/ML_BSSN::ML_BSSN_InitialADMBase2Interior: McLachlan computes the BSSN variables from the ADMBase variables. The BSSN Gamma variables are initialised using finite differencing, since they are computed from derivatives of the metric. This means that they are set only on "interior" points. 3. CCTK_INITIAL/ML_BSSN_Helper::ML_BSSN_ExtrapolateGammas: Fill the outer boundary of the Gammas by extrapolation (radial) from the interior. 4. CCTK_POSTINITIAL/MoL_PostStep/Interpolate2::Interpolate2ApplyBC: The interpatch boundaries are applied, but ONLY to points which have a wide enough stencil. Specifically, the points on both the outer boundary and the interpatch boundary are NOT set.
The extrapolation (step 3) extrapolates uninitialised points into the "corners" (those points which are both r = r_max and phi = phi_max) because the phi interpatch boundary points haven't been set yet. The interpatch boundary condition doesn't touch these points, because the interpolation stencil cannot be applied so close to the outer boundary. Hence, the points remain uninitialised.
If the interpatch BC was applied first, its stencil would include the uninitialised outer boundary points, and corrupt the interior. The extrapolation into the outer boundary would then use this corrupted value, so you would have uninitialised data in both the interior and the exterior.
A working solution is to follow the above steps, which leaves uninitialised corner points, and then apply the radial extrapolation a second time. We could probably implement this by manually scheduling ML_BSSN_Helper::ML_BSSN_ExtrapolateGammas in CCTK_POSTINITIAL after ML_BSSN_ADMBase_ApplyBCs. I think the reason I didn't do this originally was that I wanted a general solution to this problem in Kranc, and that was a harder problem. Here, we seem to be using the helper thorn to do this extrapolation anyway, so writing manual code to fix it seems like the best option. ... I have implemented this and put it on a branch of McLachlan. See https://bitbucket.org/einsteintoolkit/mclachlan/branch/ianhinder/initgammas. It compiles and passes the tests, but there is no test for this problem. If someone has an example which currently shows uninitialised points, please try it with this version and see if it is fixed.