Oh, and just to be clear, all of these three experiments kept the scheduling in CCTK_ANALYSIS intact and unchanged.
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 2:09 PM, Zach Etienne zachetie@gmail.com wrote:
Hi Ian,
Can you experiment to see what happens if you schedule your copying
routine in POSTRESTRICT? And in POSTREGRID?
Yes. I just performed the following experiments:
- Schedule myadmbaselapse=alp function in POSTRESTRICT (GLOBAL,
LOOP-LOCAL), every 64 iterations 2) Schedule myadmbaselapse=alp function in POSTREGRID (GLOBAL, LOOP-LOCAL), every 64 iterations 3) Schedule myadmbaselapse=alp function in both POSTRESTRICT and POSTRESTRICT (both GLOBAL, LOOP-LOCAL), every 64 iterations
(1) and (3) resulted in no undefined values! In (2), undefined values remained.
There remains the mystery of why calling my function in CCTK_ANALYSIS at *every* iteration results in no undefined values...
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 12:30 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 18:19, Zach Etienne zachetie@gmail.com wrote:
Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)
That tells you that alp is defined at the point it is output, but it
doesn't tell you what its state was at the time myadmbaselapse used it.
You are absolutely right. I just added a printf() statement outputting ADMBase::alp at iteration 192, at all points on the z=0 plane, inside the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did the run with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that there are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane, where many undefined values are seen.
Thus I conclude that something bad is happening to myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...
Could it be that restriction is happening after the variable is set in ANALYSIS?
Can you experiment to see what happens if you schedule your copying routine in POSTRESTRICT? And in POSTREGRID?
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 17:40, Zach Etienne zachetie@gmail.com wrote:
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn (
math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
- there's nothing complicated about the ADMBaseMcLachlanTester thorn
at all (67 lines of code, including ccl files, includes, and whitespace)
- the included parfiles run on a single desktop computer needing only
5GB of RAM
- it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if
the nan values come from there?
Absolutely! I just performed a run with ADMBaseMcLachlanTester,
setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2....
So maybe something like
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.
-- Ian Hinder http://members.aei.mpg.de/ianhin
-- Ian Hinder http://members.aei.mpg.de/ianhin