[Einstein Toolkit] #1375: Slab does not fille in outmost layer of points in mesh refinement levels
#1375: Slab does not fille in outmost layer of points in mesh refinement levels -----------------------------------+---------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: minor | Milestone: Component: EinsteinToolkit thorn | Version: development version Keywords: Slab | -----------------------------------+---------------------------------------- This happens only for mesh refined runs and on reflevel>0 ie in the buffer zones. This happens only if the "useghosts" option is not set (it is unset by default). This behaviour is technically allowed by the docs since Slab only claims to fill in ghost zones and the outermost points are not ghosts but buffer zones (though Carpet disagrees somewhat on that in that it does not count them in buffer_width).
Anyhow, this does not to my knowledge actually affect RotatingSymmetryXX which are the most prominent users, at least their poison_boundaries option does not seem to pick this up.
To test, run the attached modified slabtest.par test file whcih fails in the identity test for level 1 unless useghosts=1 is used in slabtest.c (and fails int the skip-every-second point test if it is used).
#1375: Slab does not fill in outmost layer of points in mesh refinement levels ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: minor | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: Slab ------------------------------------+---------------------------------------
#1375: Slab does not fill in outmost layer of points in mesh refinement levels ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: minor | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: Slab ------------------------------------+---------------------------------------
Comment (by eschnett):
Setting useghosts=1 in Slab is dangerous, since Slab then will use (read) ghost zones. This means that one has to synchronize before calling Slab. In multiple dimensions, points can be e.g. both symmetry and ghost zones, so that one also has to apply all symmetry conditions before synchronizing, so that these ghost zones are correct. This is in general difficult to ensure; one has to write the schedule very carefully to ensure that symmetries are applied last.
In other words: Slab would use ghost points not only to fill in ghost points, but also to fill in non-ghost points. If the ghost points have not been set (via synchronisation), non-ghost points would remain undefined.
However, points that are both symmetry and AMR ghost points (as opposed to inter-processor ghost points), cannot be set via AMR interpolation since they are too close to the boundary. That means that the symmetry condition is the only way to set them. Hence the correct solution is here: (1) synchronize first, (2) apply symmetry condition with useghosts=1. In addition to setting useghosts=1, we would also need to ensure that synchronization occurs first. Ideas?
I do not understand your argument about "technically correct". Technically, with RK3 and 3 ghost zones, a refined region is surrounded by 9 buffer zones, which are in turn surrounded by 3 ghost zones. This is Carpet's terminology, although people often call all 12 points "buffer zones".
#1375: Slab does not fill in outmost layer of points in mesh refinement levels ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: minor | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: Slab ------------------------------------+---------------------------------------
Comment (by rhaas):
Yes, my argument is based in precisely this difference in calling the outermost layer ghost points or buffer zones (naturally Carpet is the ultimate authority there so, I should really refer to them as ghost points). I should not have put this statement in the ticket though, since really it does not matter what these points are called.
The statement should have been that slab seems to properly fill in all interprocessor ghost points but not the layer of ghost points at the outermost region of a refinement level.
My actual usage case is that I want to get a full copy of all data on the grid with x->-x,y->-y,z->-z (ie all coordinates flipped), to check issues related to the center of mass. So what I have to do is: (1) make sure that I have a fully synched source grid function (2) call Slab_Transfer with useghosts=1, yes? If of course I do this in the middle of a RHS evaluation (where the ghosts and symmetry and boundary points are invalid for some time) I can just use useghosts=0 and manually clear the outermost layer to avoid nan/large values popping up in the output.
So you consider this actually something that needs to be fixed? I had not really seen it as such but only as something that should be documented.
#1375: Slab does not fill in outmost layer of points in mesh refinement levels ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: minor | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: Slab ------------------------------------+---------------------------------------
Comment (by eschnett):
I think it should be fixed. It is non-trivial to fix, since nothing in Cactus is currently keeping track of which parts of the grid are valid. If there are no bug reports, I would be inclined to wait until the schedule improvements are there, which will naturally catch and solve this.
trac@lists.einsteintoolkit.org
-
Einstein Toolkit