[Users] More Bizarre Scheduling Behavior in a Carpet Run (Bug?!)

Tue Jun 2 10:11:15 CDT 2015

Thanks for your quick feedback, Ian!

> Think about the contents of the three timelevels of myadmbaselapse.

myadmbaselapse has only one timelevel, and tags are set
'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus
prolongation should be disabled on myadmbaselapse...

Just prior to the output at, e.g., iteration 192 (the iteration when the
grid movement occurs), myadmbaselapse is set to ADMBase::alp, at all
refinement levels (due to GLOBAL,LOOP-LOCAL). Therefore, it should not
matter how often I choose to update myadmbaselapse, since the value just
prior to the file output (at iteration 192, for instance) is updated in a
GLOBAL,LOOP-LOCAL mode. So why do I see vastly different results when I
choose to update myadmbaselapse every cctk_iteration versus every 64?

-Zach

*     *     *
Zachariah Etienne
Assistant Professor of Mathematics
West Virginia University

On Tue, Jun 2, 2015 at 11:02 AM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

>
> On 2 Jun 2015, at 16:27, Zach Etienne <zachetie at gmail.com> wrote:
>
> > Hello.
> >
> > I am working on writing a diagnostic that reads in ADMBase variables
> (e.g., alp, betax, etc.) from a binary black hole simulation using
> McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a
> GLOBAL,LOOP-LOCAL context.
> >
> > Call the gridfunction quantity computed by this diagnostic,
> diag_data_gf. diag_data_gf is rather expensive to compute in general, so I
> only want it computed and output (to, e.g., IOASCII 2D data files) every 64
> iterations.
> >
> > Setting regrid_every=64, I found that the diagnostic outputs
> reasonable-looking data every 64 iterations until the first actual AMR grid
> movement (at iteration 192). At this iteration, a large number of zones
> near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are
> set to undefined values (I have set IOASCII::out3D_ghosts=no). This is
> undesirable behavior!
> >
> > Here is what I want to happen:
> > 1) every 64 iterations, the diagnostic computes diag_data_gf at all
> gridpoints on all refinement levels at CCTK_ANALYSIS (I believe
> GLOBAL,LOOP-LOCAL should do this).
> > 2) After CCTK_ANALYSIS, diag_data_gf should be output to files
> >
> > So why does my scheduling choice yield obviously wrong values near AMR
> boundaries after the grids move.
> > ********
> > Now here is where the situation becomes very weird:
> > ********
> > When I set diag_data_gf to be computed at *every* iteration, the
> undefined value problem disappears! Remember, it is being called at
> CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be
> recomputed at all points on all levels prior to file output. Where are
> these mysterious undefined values coming from?!
>
> > Upon further analysis, even at iteration 64 the diagnostic yields
> inconsistent results at all refinement levels except the finest one.
>
> >
> > ********
> > I have created a very simple thorn called ADMBaseMcLachlanTester (
> math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces
> this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of
> coding. All the thorn does is set a gridfunction called "myadmbaselapse",
> which appropriately enough, is set to ADMBase::alp at all gridpoints.
> >
> > In the thorn's par/ subdirectory, you'll find 2 parfiles:
> qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The
> former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS,
> and the latter every 64 cctk_iteration's. You will notice that the former
> parfile yields reasonable data at iteration 192 in the
> admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However,
> many nan's are produced at iteration 192 (corresponding to the first AMR
> grid movement) when using the latter parfile.
> >
> > What is causing this weird behavior? Have I uncovered a bug?
>
> If you regrid level L, Carpet will do time prolongation to fill the points
> new to the level if level L-1 (the coarser one) does not exist at the
> regridding time.  It needs to do this, since it doesn't have any grid
> points to do a purely-spatial interpolation.  Think about the contents of
> the three timelevels of myadmbaselapse.  Timelevels are cycled every
> iteration, whether you have set them or not.  Since you are setting
> timelevel 0 only every 64 iterations, the past two timelevels do not
> contain valid data for the previous two timesteps, so the points which need
> to be filled by time prolongation will contain bad data.  If you can
> partition your grids into "moving" and "fixed", and then ensure that you
> only regrid when the fixed grids exist, then you can avoid time
> interpolation when regridding, and you won't need valid data on the past
> timelevels.  There is also a parameter
>
>         Carpet::time_interpolation_during_regridding (
> https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701cf8d1c5/Carpet/param.ccl?at=master#cl-586
> )
>
> but I'm not sure exactly what it does.  A quick grep of the Carpet source
> should reveal this.  It might be helpful.  But the best option is probably
> to just arrange your regridding frequency so that you don't need to
> interpolate in time.  i.e., if your first fixed grid exists every 32
> iterations, then only regrid on multiples of 32 iterations.
>
> There is another parameter, if you are using CarpetRegrid2,
>
>         CarpetRegrid2::freeze_unaligned_parent_levels
>         Do not change refinement levels where the parent does not exist at
> this time
>
> (
> https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701cf8d1c5/CarpetRegrid2/param.ccl?at=master#cl-26
> )
>
> but I think this will stop the regridding from actually happening, so you
> would have to be careful that you do eventually regrid on an iteration when
> the grid is aligned with its parent.
>
> --
> Ian Hinder
> http://members.aei.mpg.de/ianhin
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20150602/5574dfe4/attachment-0001.html 