[Users] Evolving grid arrays using MoL

[Ian Hinder]

On 24 Sep 2013, at 11:34, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

> 
> On 23 Sep 2013, at 18:06, Peter Diener <diener at cct.lsu.edu> wrote:
> 
>> Hi,
>> 
>> No, you're right. The examples I mentioned are using either
>> uni-grid or multi-patch without AMR. I didn't think of the complication of adding AMR to the mix.
> 
> So unless anything has changed, the latest information I have is from Erik's email from September 2007:
> 
>> On 28 Sep 2007, at 18:10, Erik Schnetter <schnetter at cct.lsu.edu> wrote:
>> 
>>> I am now evolving geodesics with the following work-around:
>>> 
>>> I use MoL.  Each time the RHS is calculated, I check whether the current refinement level is the finest level.  If so, the RHS is calculated, otherwise the RHS is set to zero.
>>> 
>>> The current refinement level is accessed by
>>> 
>>> interface.ccl:
>>> USES INCLUDE: carpet.hh
>>> 
>>> #include <carpet.hh>
>>> 
>>> if (Carpet::reflevel == Carpet::reflevels - 1) // do calculation
>>> 
>>> This works only from C++.  The RHS calculating routine has to be scheduled in level mode.

I am using this method, and I have convinced myself that it is correct.  The following facts are pertinent:

	• Carpet loops through all iterations (of the finest possible grid), and within each iteration loops through all refinement levels from coarse to fine
	• For each iteration, Carpet only cycles the timelevels of grid arrays for the first (coarsest) refinement level that exists at that iteration.  This means the timelevels are only cycled once, which is good.
	• MoL copies grid arrays from the past timelevel to the current timelevel on every refinement level

These facts combine to mean that no matter what RHS is used on the coarse grids, the correct past timelevel data is used to set the initial data for the iteration for MoL on the fine grid.  I believe that you can compute the RHSs on all levels, without the check for the current level being the finest, and you still get the answer that was computed for the fine grid.  This is because the values computed on the coarser grids are always overwritten (by MoL_InitialCopy) when the finer grid is evolved on that iteration.  You could even set the RHSs to poison on the coarser grids (assuming that MoL can evolve poison without becoming slow or crashing).  The latter is probably equivalent to scheduling the RHS routine in global-late, as I think Erik suggested in another post, as global-late routines are evolved only on the fine grid.

I suspect that this scheme will work if levels are added or removed during the simulation, but I haven't thought much about that case.

All the above ignores any potential interactions with the interpolator.  I am currently calling the RHS routine in level mode, and setting the interpolator option "want_global_mode".  There is a comment in CarpetInterp which says that this causes the interpolation to happen in global mode, i.e. with data from all timelevels.  This seems to work; interpolation points not on the current level seem to give reasonable answers.  

If I interpolate a variable within MoL_CalcRHS, I want the value at the current (MoL-set) value of cctk_time.  This would be t_n, t_{n+1/2} or t_{n+1} for RK4.  It looks like CarpetInterp uses the value of cctk_time to determine the time at which to interpolate.  Interpolation performed on a coarse grid cannot give correct values from the fine grid if the interpolation point is in the fine grid, since the fine grid won't have been evolved to the required time yet.  What happens in that situation?  If Carpet aborts with an "extrapolation in time" error, then this means we have no choice but to evolve on the finest grid, which could be very inefficient.

I think we want to perform the interpolation in global-late (i.e. after all levels have been evolved), but not necessarily every iteration. Maybe we can fake this by setting the RHS to 0 on the iterations where we don't want to integrate, and setting it to rhs * dT/dt on the iterations where we do want to integrate, so that MoL will effectively use the correspondingly larger step size.

So the summary would be:

	• Schedule the RHS routine in global-late, so it is only run on the finest level (and hence once all the others have been evolved).  This probably causes MoL to integrate poison while on the other levels, but this should not make it into the output as the finest level will override the result.
	• Use the interpolator parameter "want_global_mode", as this allows the interpolation to use data from all refinement levels.
	• If you want to evolve less frequently than every fine-grid iteration, add the required logic to your RHS routine, and multiply the RHS by the corresponding factor.

Opinions?

-- 
Ian Hinder
http://numrel.aei.mpg.de/people/hinder

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20131104/8a0d059c/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20131104/8a0d059c/attachment.bin