[Users] Evolving grid arrays using MoL

[Ian Hinder]

On 7 Oct 2013, at 19:34, Erik Schnetter <schnetter at cct.lsu.edu> wrote:

> On 2013-10-07, at 12:47 , Ian Hinder <ian.hinder at aei.mpg.de> wrote:
> 
>> On 28 Sep 2013, at 16:17, Erik Schnetter <schnetter at gmail.com> wrote:
>> 
>>> Yes, this (i.e. the work-around of scheduling the RHS in level mode, and then only actually executing the routine if this corresponds to global mode) is how I made things work. I haven't tried this recently, though. Maybe a test case would be in order...
>>> 
>>> Rather straightforward changes to MoL should allow scheduling the RHS in global mode as well, which would then work elegantly. This would probably require to duplicate MoL's integration logic, so that grid functions are integrated in level mode while grid arrays are integrated in global mode.
>> 
>> Is the problem that the grid arrays are not available in global mode, but they are available in level mode?  Or is there some other reason that global mode cannot be used?  When I try to call the interpolator in MoL_CalcRHS in level mode, I get
>> 
>> Assertion failed: (not (not want_global_mode and num_time_derivs.AT(m) == 0 and need_time_interp.AT(m))), function interpolate_components, file /Users/ian/Cactus/arrangements/Carpet/CarpetInterp/src/interp.cc, line 1353.
>> 
>> This is with InterpNumTimelevels = 1 in the interpolator parameters.  I dug around in interp.cc, and came across the interpolation parameter want_global_mode.  Setting this to 1 seems to allow the interpolation to succeed, but I don't know if this will break in certain situations.  What does this interpolator parameter do?
>> 
>> My problem with registration turned out to be that I was trying to register MoL for grid SCALARs, not grid ARRAYs.  A vector of grid arrays seems to work.
> 
> 
> Grid arrays are available in global mode (and also in level, singlemap, and local mode).
> 
> Global mode cannot be used because MoL will still try to evolve them in level mode, i.e. more often. One needs to ensure that grid arrays have a non-zero RHS exactly on the finest grid. Global mode may be equivalent to the coarsest grid, which would then evolve with the wrong step size.

What does "equivalent to the coarsest grid" mean?  Global mode routines are not called only every coarse grid timestep; I think they are called every fine grid iteration, when that level exists.  As Frank pointed out, this could be problematic.  

How/when are grid array timelevels rotated?  I suspect this happens every fine grid iteration.  What happens if a new finest grid is activated during a simulation?  Does the effective dt of the grid array integration now become smaller?  This means that the timelevels will have a nonuniform dt.  Can MoL handle this?

Maybe there should be a concept of a grid array "belonging" to a specific refinement level, in the sense that it will only be "updated" when that refinement level exists.  This means its timelevels will be rotated only then, and functions that want to access it will only be allowed to access it at those times.  This would also allow integrations on a time step larger than the finest grid; at the moment, if I want to integrate something on the coarse grid, I have to do so with a timestep corresponding to the finest grid, which is obviously not very efficient.

The concept of having grid arrays separate from the normal grid functions is good in a "spatial" sense, but since Cactus treats timelevels globally, we have to deal with the issue of time refinement for grid arrays.  An alternative would be to have a separate time hierarchy notion for grid arrays.  Each grid array would have its own "dt".  This might not even be commensurate with the gf time refinement factors.  You would need a new way of scheduling a function "at" the time of a specific array (or group of arrays) in global mode, and during this call, you would likely need time interpolation if you wanted to access grid functions.  I think this would be the best way of handling time stepping for grid arrays, but I think it would need some fairly serious changes to Carpet, and maybe also to parts of Cactus.  Maybe attaching grid arrays to specific refinement levels is easier, and likely sufficient.  

> Yes, one cannot interpolate in level mode if this requires data from coarser levels. This is just what level mode means -- access data from the current refinement level only.
> 
> Is there a mode "global-finest"? This should work. Or maybe "global-late" for evolution, since finest grid are evolved last? Yes, I think this should work.

Do these exist at the moment, or would they need to be added?

> Grid scalars are just a funny name for 0-dimensional grid arrays. If MoL doesn't support this, then it would be trivial to add -- the only difference is that they are called "scalar" instead of "array".

I'm not sure why MoL needs to even know about the group type (gf, scalar or array).  I thought that Cactus flesh API was designed so that you can ask for a pointer to the variable, find the number of dimensions and size of the array, and loop over it, all without knowing whether it is a grid function or an array?  Why does MoL know the difference?  Knowing the difference means it has a lot of duplicated code, which is bad.

What does the interpolator parameter want_global_mode do?  Should I be using it?

-- 
Ian Hinder
http://numrel.aei.mpg.de/people/hinder

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20131007/a18b13cb/attachment-0001.bin