[Users] More Bizarre Scheduling Behavior in a Carpet Run (Bug?!)

[Ian Hinder]

On 2 Jun 2015, at 16:27, Zach Etienne <zachetie at gmail.com> wrote:

> Hello.
> 
> I am working on writing a diagnostic that reads in ADMBase variables (e.g., alp, betax, etc.) from a binary black hole simulation using McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a GLOBAL,LOOP-LOCAL context. 
> 
> Call the gridfunction quantity computed by this diagnostic, diag_data_gf. diag_data_gf is rather expensive to compute in general, so I only want it computed and output (to, e.g., IOASCII 2D data files) every 64 iterations.
> 
> Setting regrid_every=64, I found that the diagnostic outputs reasonable-looking data every 64 iterations until the first actual AMR grid movement (at iteration 192). At this iteration, a large number of zones near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are set to undefined values (I have set IOASCII::out3D_ghosts=no). This is undesirable behavior!
> 
> Here is what I want to happen:
> 1) every 64 iterations, the diagnostic computes diag_data_gf at all gridpoints on all refinement levels at CCTK_ANALYSIS (I believe GLOBAL,LOOP-LOCAL should do this).
> 2) After CCTK_ANALYSIS, diag_data_gf should be output to files
> 
> So why does my scheduling choice yield obviously wrong values near AMR boundaries after the grids move.
> ********
> Now here is where the situation becomes very weird:
> ********
> When I set diag_data_gf to be computed at *every* iteration, the undefined value problem disappears! Remember, it is being called at CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be recomputed at all points on all levels prior to file output. Where are these mysterious undefined values coming from?!

> Upon further analysis, even at iteration 64 the diagnostic yields inconsistent results at all refinement levels except the finest one.

> 
> ********
> I have created a very simple thorn called ADMBaseMcLachlanTester (math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of coding. All the thorn does is set a gridfunction called "myadmbaselapse", which appropriately enough, is set to ADMBase::alp at all gridpoints.
> 
> In the thorn's par/ subdirectory, you'll find 2 parfiles: qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS, and the latter every 64 cctk_iteration's. You will notice that the former parfile yields reasonable data at iteration 192 in the admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However, many nan's are produced at iteration 192 (corresponding to the first AMR grid movement) when using the latter parfile.
> 
> What is causing this weird behavior? Have I uncovered a bug?

If you regrid level L, Carpet will do time prolongation to fill the points new to the level if level L-1 (the coarser one) does not exist at the regridding time.  It needs to do this, since it doesn't have any grid points to do a purely-spatial interpolation.  Think about the contents of the three timelevels of myadmbaselapse.  Timelevels are cycled every iteration, whether you have set them or not.  Since you are setting timelevel 0 only every 64 iterations, the past two timelevels do not contain valid data for the previous two timesteps, so the points which need to be filled by time prolongation will contain bad data.  If you can partition your grids into "moving" and "fixed", and then ensure that you only regrid when the fixed grids exist, then you can avoid time interpolation when regridding, and you won't need valid data on the past timelevels.  There is also a parameter

	Carpet::time_interpolation_during_regridding (https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701cf8d1c5/Carpet/param.ccl?at=master#cl-586)

but I'm not sure exactly what it does.  A quick grep of the Carpet source should reveal this.  It might be helpful.  But the best option is probably to just arrange your regridding frequency so that you don't need to interpolate in time.  i.e., if your first fixed grid exists every 32 iterations, then only regrid on multiples of 32 iterations.

There is another parameter, if you are using CarpetRegrid2,

	CarpetRegrid2::freeze_unaligned_parent_levels 
	Do not change refinement levels where the parent does not exist at this time

(https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701cf8d1c5/CarpetRegrid2/param.ccl?at=master#cl-26)

but I think this will stop the regridding from actually happening, so you would have to be careful that you do eventually regrid on an iteration when the grid is aligned with its parent.

-- 
Ian Hinder
http://members.aei.mpg.de/ianhin