Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)

> That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it.  

You are absolutely right. I just added a printf() statement outputting ADMBase::alp at iteration 192, at all points on the z=0 plane, inside the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did the run with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that there are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane, where many undefined values are seen.

Thus I conclude that something bad is happening to myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...

-Zach

*     *     *
Zachariah Etienne
Assistant Professor of Mathematics
West Virginia University

On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder <ian.hinder@aei.mpg.de> wrote:

On 2 Jun 2015, at 17:40, Zach Etienne <zachetie@gmail.com> wrote:

> Hi Frank,
>
> Thanks for your feedback.
> > You use the ADMBase variables, right?
> My ADMBaseMcLachlanTester thorn (math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
> 1) there's nothing complicated about the ADMBaseMcLachlanTester thorn at all (67 lines of code, including ccl files, includes, and whitespace)
> 2) the included parfiles run on a single desktop computer needing only 5GB of RAM
> 3) it takes only ~15 mins to evolve to timestep 192
>
> > Could you output the variables your analysis depends on, and see if the nan values come from there?
> Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
>
> So myadmbaselapse is not undefined because ADMBase::alp is undefined.

That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it.  I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point.  http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2.html#x4-139000A2

So maybe something like

CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")

and you should get your 2D output at the point the variable is set.  You could do the same for alp, to see what it is being set from.  Or you could use printf.