<div dir="ltr">Oh, and just to be clear, all of these three experiments kept the scheduling in CCTK_ANALYSIS intact and unchanged.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>-Zach</div><div><br></div>* * *<br>Zachariah Etienne<br>Assistant Professor of Mathematics<div>West Virginia University<br></div></div></div></div>
<br><div class="gmail_quote">On Tue, Jun 2, 2015 at 2:09 PM, Zach Etienne <span dir="ltr"><<a href="mailto:zachetie@gmail.com" target="_blank">zachetie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ian,<div><br></div><div><span class=""><div style="font-size:12.8000001907349px">> Can you experiment to see what happens if you schedule your copying routine in POSTRESTRICT? And in POSTREGRID?</div><div style="font-size:12.8000001907349px"><br></div></span><div style="font-size:12.8000001907349px">Yes. I just performed the following experiments:</div><div style="font-size:12.8000001907349px">1) Schedule <span style="font-size:12.8000001907349px">myadmbaselapse=alp function</span><span style="font-size:12.8000001907349px"> in </span><span style="font-size:12.8000001907349px">POSTRESTRICT (GLOBAL, LOOP-LOCAL), every 64 iterations</span></div><div><span style="font-size:12.8000001907349px">2) Schedule </span><span style="font-size:12.8000001907349px">myadmbaselapse=alp function </span><span style="font-size:12.8000001907349px">in </span><span style="font-size:12.8000001907349px">POSTREGRID </span><span style="font-size:12.8000001907349px">(GLOBAL, LOOP-LOCAL), every 64 iterations</span></div><div><span style="font-size:12.8000001907349px">3) Schedule </span><span style="font-size:12.8000001907349px">myadmbaselapse=alp function </span><span style="font-size:12.8000001907349px">in both </span><span style="font-size:12.8000001907349px">POSTRESTRICT and </span><span style="font-size:12.8000001907349px">POSTRESTRICT (both </span><span style="font-size:12.8000001907349px">GLOBAL, LOOP-LOCAL), every 64 iterations</span></div><div style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px"><br></span></div><div style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">(1) and (3) resulted in no undefined values! In (2), undefined values remained.</span></div><div style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px"><br></span></div><div style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">There remains the mystery of why calling my function in CCTK_ANALYSIS at *every* iteration results in no undefined values...</span></div><div style="font-size:12.8000001907349px"></div></div><div class="gmail_extra"><span class=""><br clear="all"><div><div><div dir="ltr"><div>-Zach</div><div><br></div>* * *<br>Zachariah Etienne<br>Assistant Professor of Mathematics<div>West Virginia University<br></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Tue, Jun 2, 2015 at 12:30 PM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span><div>On 2 Jun 2015, at 18:19, Zach Etienne <<a href="mailto:zachetie@gmail.com" target="_blank">zachetie@gmail.com</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)<br><div><br></div><div>> <span style="font-size:12.8000001907349px">That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. </span><span style="font-size:12.8000001907349px"> </span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">You are absolutely right. I just added a printf() statement outputting </span><span style="font-size:12.8000001907349px">ADMBase::alp</span><span style="font-size:12.8000001907349px"> at iteration 192, at all points on the z=0 plane,</span><span style="font-size:12.8000001907349px"> inside the </span>ADMBaseMcLachlanTester<span style="font-size:12.8000001907349px"> loop that sets </span><span style="font-size:12.8000001907349px">myadmbaselapse=alp. Then I</span><span style="font-size:12.8000001907349px"> re-did the run </span>with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that t<span style="font-size:12.8000001907349px">here are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the </span><span style="font-size:12.8000001907349px">myadmbaselapse </span><span style="font-size:12.8000001907349px">IOASCII 2D file in the z=0 plane, where many undefined values are seen.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Thus I conclude that something bad is happening to </span><span style="font-size:12.8000001907349px">myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...</span></div></div></blockquote><div><br></div></span><div>Could it be that restriction is happening after the variable is set in ANALYSIS?</div><div><br></div><div>Can you experiment to see what happens if you schedule your copying routine in POSTRESTRICT? And in POSTREGRID?</div><div><div><div><br></div><br><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div>-Zach</div><div><br></div>* * *<br>Zachariah Etienne<br>Assistant Professor of Mathematics<div>West Virginia University<br></div></div></div>
<br><div class="gmail_quote">On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
On 2 Jun 2015, at 17:40, Zach Etienne <<a href="mailto:zachetie@gmail.com" target="_blank">zachetie@gmail.com</a>> wrote:<br>
<br>
> Hi Frank,<br>
><br>
> Thanks for your feedback.<br>
> > You use the ADMBase variables, right?<br>
> My ADMBaseMcLachlanTester thorn (<a href="http://math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz" target="_blank">math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz</a>) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as<br>
> 1) there's nothing complicated about the ADMBaseMcLachlanTester thorn at all (67 lines of code, including ccl files, includes, and whitespace)<br>
> 2) the included parfiles run on a single desktop computer needing only 5GB of RAM<br>
> 3) it takes only ~15 mins to evolve to timestep 192<br>
><br>
> > Could you output the variables your analysis depends on, and see if the nan values come from there?<br>
> Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...<br>
><br>
> So myadmbaselapse is not undefined because ADMBase::alp is undefined.<br>
<br>
</span>That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. <a href="http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2.html#x4-139000A2" target="_blank">http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2.html#x4-139000A2</a><br>
<br>
So maybe something like<br>
<br>
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")<br>
<br>
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.<br>
<div><br>
--<br>
Ian Hinder<br>
<a href="http://members.aei.mpg.de/ianhin" target="_blank">http://members.aei.mpg.de/ianhin</a><br>
<br>
</div></blockquote></div><br></div></div>
</blockquote></div></div></div><div><div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>-- </div><div>Ian Hinder</div><div><a href="http://members.aei.mpg.de/ianhin" target="_blank">http://members.aei.mpg.de/ianhin</a></div></div></div></div></div>
</div>
<br></div></div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>