Hello.
I am working on writing a diagnostic that reads in ADMBase variables (e.g., alp, betax, etc.) from a binary black hole simulation using McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a GLOBAL,LOOP-LOCAL context.
Call the gridfunction quantity computed by this diagnostic, diag_data_gf. diag_data_gf is rather expensive to compute in general, so I only want it computed and output (to, e.g., IOASCII 2D data files) every 64 iterations.
Setting regrid_every=64, I found that the diagnostic outputs reasonable-looking data every 64 iterations until the first actual AMR grid movement (at iteration 192). At this iteration, a large number of zones near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are set to undefined values (I have set IOASCII::out3D_ghosts=no). This is undesirable behavior!
Here is what I want to happen: 1) every 64 iterations, the diagnostic computes diag_data_gf at all gridpoints on all refinement levels at CCTK_ANALYSIS (I believe GLOBAL,LOOP-LOCAL should do this). 2) After CCTK_ANALYSIS, diag_data_gf should be output to files
So why does my scheduling choice yield obviously wrong values near AMR boundaries after the grids move.
******** Now here is where the situation becomes very weird: ******** When I set diag_data_gf to be computed at *every* iteration, the undefined value problem disappears! Remember, it is being called at CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be recomputed at all points on all levels prior to file output. Where are these mysterious undefined values coming from?!
Upon further analysis, even at iteration 64 the diagnostic yields inconsistent results at all refinement levels except the finest one.
******** I have created a very simple thorn called ADMBaseMcLachlanTester ( math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of coding. All the thorn does is set a gridfunction called "myadmbaselapse", which appropriately enough, is set to ADMBase::alp at all gridpoints.
In the thorn's par/ subdirectory, you'll find 2 parfiles: qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS, and the latter every 64 cctk_iteration's. You will notice that the former parfile yields reasonable data at iteration 192 in the admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However, many nan's are produced at iteration 192 (corresponding to the first AMR grid movement) when using the latter parfile.
What is causing this weird behavior? Have I uncovered a bug?
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University