[Users] More Bizarre Scheduling Behavior in a Carpet Run (Bug?!)

[Zach Etienne]

Forgot to mention something: This run has only 7 refinement levels in total
(i.e., the coarsest level, plus 6 additional levels of refinement), so
every 64 timesteps, all levels should be updated.

-Zach

*     *     *
Zachariah Etienne
Assistant Professor of Mathematics
West Virginia University

On Tue, Jun 2, 2015 at 11:11 AM, Zach Etienne <zachetie at gmail.com> wrote:

> Thanks for your quick feedback, Ian!
>
> > Think about the contents of the three timelevels of myadmbaselapse.
>
> myadmbaselapse has only one timelevel, and tags are set
> 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus
> prolongation should be disabled on myadmbaselapse...
>
> Just prior to the output at, e.g., iteration 192 (the iteration when the
> grid movement occurs), myadmbaselapse is set to ADMBase::alp, at all
> refinement levels (due to GLOBAL,LOOP-LOCAL). Therefore, it should not
> matter how often I choose to update myadmbaselapse, since the value just
> prior to the file output (at iteration 192, for instance) is updated in a
> GLOBAL,LOOP-LOCAL mode. So why do I see vastly different results when I
> choose to update myadmbaselapse every cctk_iteration versus every 64?
>
>
> -Zach
>
> *     *     *
> Zachariah Etienne
> Assistant Professor of Mathematics
> West Virginia University
>
> On Tue, Jun 2, 2015 at 11:02 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
>
>>
>> On 2 Jun 2015, at 16:27, Zach Etienne <zachetie at gmail.com> wrote:
>>
>> > Hello.
>> >
>> > I am working on writing a diagnostic that reads in ADMBase variables
>> (e.g., alp, betax, etc.) from a binary black hole simulation using
>> McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a
>> GLOBAL,LOOP-LOCAL context.
>> >
>> > Call the gridfunction quantity computed by this diagnostic,
>> diag_data_gf. diag_data_gf is rather expensive to compute in general, so I
>> only want it computed and output (to, e.g., IOASCII 2D data files) every 64
>> iterations.
>> >
>> > Setting regrid_every=64, I found that the diagnostic outputs
>> reasonable-looking data every 64 iterations until the first actual AMR grid
>> movement (at iteration 192). At this iteration, a large number of zones
>> near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are
>> set to undefined values (I have set IOASCII::out3D_ghosts=no). This is
>> undesirable behavior!
>> >
>> > Here is what I want to happen:
>> > 1) every 64 iterations, the diagnostic computes diag_data_gf at all
>> gridpoints on all refinement levels at CCTK_ANALYSIS (I believe
>> GLOBAL,LOOP-LOCAL should do this).
>> > 2) After CCTK_ANALYSIS, diag_data_gf should be output to files
>> >
>> > So why does my scheduling choice yield obviously wrong values near AMR
>> boundaries after the grids move.
>> > ********
>> > Now here is where the situation becomes very weird:
>> > ********
>> > When I set diag_data_gf to be computed at *every* iteration, the
>> undefined value problem disappears! Remember, it is being called at
>> CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be
>> recomputed at all points on all levels prior to file output. Where are
>> these mysterious undefined values coming from?!
>>
>> > Upon further analysis, even at iteration 64 the diagnostic yields
>> inconsistent results at all refinement levels except the finest one.
>>
>> >
>> > ********
>> > I have created a very simple thorn called ADMBaseMcLachlanTester (
>> math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces
>> this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of
>> coding. All the thorn does is set a gridfunction called "myadmbaselapse",
>> which appropriately enough, is set to ADMBase::alp at all gridpoints.
>> >
>> > In the thorn's par/ subdirectory, you'll find 2 parfiles:
>> qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The
>> former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS,
>> and the latter every 64 cctk_iteration's. You will notice that the former
>> parfile yields reasonable data at iteration 192 in the
>> admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However,
>> many nan's are produced at iteration 192 (corresponding to the first AMR
>> grid movement) when using the latter parfile.
>> >
>> > What is causing this weird behavior? Have I uncovered a bug?
>>
>> If you regrid level L, Carpet will do time prolongation to fill the
>> points new to the level if level L-1 (the coarser one) does not exist at
>> the regridding time.  It needs to do this, since it doesn't have any grid
>> points to do a purely-spatial interpolation.  Think about the contents of
>> the three timelevels of myadmbaselapse.  Timelevels are cycled every
>> iteration, whether you have set them or not.  Since you are setting
>> timelevel 0 only every 64 iterations, the past two timelevels do not
>> contain valid data for the previous two timesteps, so the points which need
>> to be filled by time prolongation will contain bad data.  If you can
>> partition your grids into "moving" and "fixed", and then ensure that you
>> only regrid when the fixed grids exist, then you can avoid time
>> interpolation when regridding, and you won't need valid data on the past
>> timelevels.  There is also a parameter
>>
>>         Carpet::time_interpolation_during_regridding (
>> https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701cf8d1c5/Carpet/param.ccl?at=master#cl-586
>> )
>>
>> but I'm not sure exactly what it does.  A quick grep of the Carpet source
>> should reveal this.  It might be helpful.  But the best option is probably
>> to just arrange your regridding frequency so that you don't need to
>> interpolate in time.  i.e., if your first fixed grid exists every 32
>> iterations, then only regrid on multiples of 32 iterations.
>>
>> There is another parameter, if you are using CarpetRegrid2,
>>
>>         CarpetRegrid2::freeze_unaligned_parent_levels
>>         Do not change refinement levels where the parent does not exist
>> at this time
>>
>> (
>> https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701cf8d1c5/CarpetRegrid2/param.ccl?at=master#cl-26
>> )
>>
>> but I think this will stop the regridding from actually happening, so you
>> would have to be careful that you do eventually regrid on an iteration when
>> the grid is aligned with its parent.
>>
>> --
>> Ian Hinder
>> http://members.aei.mpg.de/ianhin
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20150602/8153e1d9/attachment.html