[ET Trac] [Einstein Toolkit] #2030: Multi-block boundaries leave uninitialized boundary points

[Einstein Toolkit]

#2030: Multi-block boundaries leave uninitialized boundary points
------------------------------------+---------------------------------------
  Reporter:  eschnett               |       Owner:                     
      Type:  defect                 |      Status:  new                
  Priority:  major                  |   Milestone:                     
 Component:  EinsteinToolkit thorn  |     Version:  development version
Resolution:                         |    Keywords:                     
------------------------------------+---------------------------------------

Comment (by hinder):

 When syncing ghost zones (for interprocess communication), each ghost
 point can be updated in precisely one way, which is to set its value to
 the value held by the process which "owns" the real point.  i.e., we have
 only one "boundary condition" which can be applied to that point.  A
 "standard" interpatch boundary point is the same; in the usual case, there
 is a single component on a single process which owns the point, and the
 boundary point is filled by interpolation from the neighbouring points in
 that component.  Similarly, a "standard" outer boundary point can be
 filled by applying the physical boundary condition.  However, a problem
 arises when a point is BOTH an outer boundary point and an interpatch
 boundary point.  This would happen for a point at r = r_max and phi =
 phi_max.

 I think this problem only happens during initialisation, not evolution.
 The steps are these:

 1. CCTK_INITIAL/ADMBase_InitialData: TwoPunctures initialises the ADMBase
 metric variables pointwise at every point on the grid.
 2. CCTK_INITIAL/ML_BSSN::ML_BSSN_InitialADMBase2Interior: McLachlan
 computes the BSSN variables from the ADMBase variables.  The BSSN Gamma
 variables are initialised using finite differencing, since they are
 computed from derivatives of the metric.  This means that they are set
 only on "interior" points.
 3. CCTK_INITIAL/ML_BSSN_Helper::ML_BSSN_ExtrapolateGammas: Fill the outer
 boundary of the Gammas by extrapolation (radial) from the interior.
 4. CCTK_POSTINITIAL/MoL_PostStep/Interpolate2::Interpolate2ApplyBC: The
 interpatch boundaries are applied, but ONLY to points which have a wide
 enough stencil. Specifically, the points on both the outer boundary and
 the interpatch boundary are NOT set.

 The extrapolation (step 3) extrapolates uninitialised points into the
 "corners" (those points which are both r = r_max and phi = phi_max)
 because the phi interpatch boundary points haven't been set yet.  The
 interpatch boundary condition doesn't touch these points, because the
 interpolation stencil cannot be applied so close to the outer boundary.
 Hence, the points remain uninitialised.

 If the interpatch BC was applied first, its stencil would include the
 uninitialised outer boundary points, and corrupt the interior.  The
 extrapolation into the outer boundary would then use this corrupted value,
 so you would have uninitialised data in both the interior and the
 exterior.

 A working solution is to follow the above steps, which leaves
 uninitialised corner points, and then apply the radial extrapolation a
 second time. We could probably implement this by manually scheduling
 ML_BSSN_Helper::ML_BSSN_ExtrapolateGammas in CCTK_POSTINITIAL after
 ML_BSSN_ADMBase_ApplyBCs.  I think the reason I didn't do this originally
 was that I wanted a general solution to this problem in Kranc, and that
 was a harder problem.  Here, we seem to be using the helper thorn to do
 this extrapolation anyway, so writing manual code to fix it seems like the
 best option. ... I have implemented this and put it on a branch of
 McLachlan.  See
 https://bitbucket.org/einsteintoolkit/mclachlan/branch/ianhinder/initgammas.
 It compiles and passes the tests, but there is no test for this problem.
 If someone has an example which currently shows uninitialised points,
 please try it with this version and see if it is fixed.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/2030#comment:4>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit