[Users] More Bizarre Scheduling Behavior in a Carpet Run (Bug?!)

[Ian Hinder]

On 2 Jun 2015, at 17:11, Zach Etienne <zachetie at gmail.com> wrote:

> Thanks for your quick feedback, Ian!
> 
> > Think about the contents of the three timelevels of myadmbaselapse. 
> 
> myadmbaselapse has only one timelevel, and tags are set 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus prolongation should be disabled on myadmbaselapse...
> 
> Just prior to the output at, e.g., iteration 192 (the iteration when the grid movement occurs), myadmbaselapse is set to ADMBase::alp, at all refinement levels (due to GLOBAL,LOOP-LOCAL). Therefore, it should not matter how often I choose to update myadmbaselapse, since the value just prior to the file output (at iteration 192, for instance) is updated in a GLOBAL,LOOP-LOCAL mode. So why do I see vastly different results when I choose to update myadmbaselapse every cctk_iteration versus every 64?

Pfft.  And I thought it was going to be easy...  Have you checked the logic for "updating every 64"?  Depending on where you schedule, cctk_iteration might be one different to a multiple of 64.  Though I thought that this is only off-by-one in EVOL, and since you are scheduling in ANALYSIS, it should be correct.  If you add a print statement to the updating routine, can you verify that it is being called on the correct refinement level at the iterations you are expecting?

This off-by-one has always confused me, but Erik explained it at one point and it made sense. I think it was needed for having a monotonically-increasing iteration counter.  

Another thought: you said that the finest grid looked ok, but the coarser grids had problems.  This points in the direction of restriction, which is not called on the finest grid.  Which region of the coarse grid looks wrong?  Could it be the points which are restricted, or are within a few points of the refinement boundary?  I believe restriction is also not called for routines in ANALYSIS.

What happens if you schedule your calculation routine in POSTREGRID?  And maybe POSTRESTRICT?

From the schedule output, it looks like regridding happens before evolution.  You have told it not to prolongate, so maybe this has the effect of not prolongating during regridding.  Hence the values will be uninitialised after regridding.  After REGRID, POSTREGRID is run, in which McLachlan updates ADMBase::alp from ML_BSSN::alpha.  I believe that if you have regrid_every = 64, then the grids are modified when performing the EVOL step which produces the data for iteration 64.  Could it be that your routine is called when the ADMBase data is wrong?  It looks like it would be helpful to find out whether your variable is correct when you set it, and then corrupted afterwards, or whether it is being set wrongly in the first place, e.g. because the ADMBase lapse is bad at that time.

I'm not really sure what the problem is, but maybe some of the ideas above will help?  Maybe this is related to the order in which refinement levels are traversed in POSTRESTRICT.  We had trouble with that before.  I'm not an expert in the scheduling modes.  What exactly does GLOBAL,LOOP-LOCAL mean?

-- 
Ian Hinder
http://members.aei.mpg.de/ianhin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20150602/78403b6e/attachment-0001.html