[Users] More Bizarre Scheduling Behavior in a Carpet Run (Bug?!)

[Zach Etienne]

Oh, and just to be clear, all of these three experiments kept the
scheduling in CCTK_ANALYSIS intact and unchanged.

-Zach

*     *     *
Zachariah Etienne
Assistant Professor of Mathematics
West Virginia University

On Tue, Jun 2, 2015 at 2:09 PM, Zach Etienne <zachetie at gmail.com> wrote:

> Hi Ian,
>
> > Can you experiment to see what happens if you schedule your copying
> routine in POSTRESTRICT?  And in POSTREGRID?
>
> Yes. I just performed the following experiments:
> 1) Schedule myadmbaselapse=alp function in POSTRESTRICT (GLOBAL,
> LOOP-LOCAL), every 64 iterations
> 2) Schedule myadmbaselapse=alp function in POSTREGRID (GLOBAL,
> LOOP-LOCAL), every 64 iterations
> 3) Schedule myadmbaselapse=alp function in both POSTRESTRICT and POSTRESTRICT
> (both GLOBAL, LOOP-LOCAL), every 64 iterations
>
> (1) and (3) resulted in no undefined values! In (2), undefined values
> remained.
>
> There remains the mystery of why calling my function in CCTK_ANALYSIS at
> *every* iteration results in no undefined values...
>
> -Zach
>
> *     *     *
> Zachariah Etienne
> Assistant Professor of Mathematics
> West Virginia University
>
> On Tue, Jun 2, 2015 at 12:30 PM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
>
>>
>> On 2 Jun 2015, at 18:19, Zach Etienne <zachetie at gmail.com> wrote:
>>
>> Thanks for getting back to me, Ian, and thanks for agreeing this does not
>> appear to be a trivial problem. (I've been battling it on and off for days.)
>>
>> > That tells you that alp is defined at the point it is output, but it
>> doesn't tell you what its state was at the time myadmbaselapse used it.
>>
>> You are absolutely right. I just added a printf() statement outputting
>> ADMBase::alp at iteration 192, at all points on the z=0 plane, inside
>> the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did
>> the run with ADMBaseMcLachlanTester. After analyzing the output, I can
>> confirm that there are no undefined (nan) values at iteration 192, and
>> alp values from this printf() statement lie in the very reasonable range of
>> 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the
>> data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane,
>> where many undefined values are seen.
>>
>> Thus I conclude that something bad is happening to myadmbaselapse after
>> being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but
>> before the data are written to file at the same iteration, causing
>> perfectly reasonable values to suddenly become undefined. For some reason,
>> the same behavior is not observed in ADMBase::alp...
>>
>>
>> Could it be that restriction is happening after the variable is set in
>> ANALYSIS?
>>
>> Can you experiment to see what happens if you schedule your copying
>> routine in POSTRESTRICT?  And in POSTREGRID?
>>
>>
>>
>> -Zach
>>
>> *     *     *
>> Zachariah Etienne
>> Assistant Professor of Mathematics
>> West Virginia University
>>
>> On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder <ian.hinder at aei.mpg.de>
>> wrote:
>>
>>>
>>> On 2 Jun 2015, at 17:40, Zach Etienne <zachetie at gmail.com> wrote:
>>>
>>> > Hi Frank,
>>> >
>>> > Thanks for your feedback.
>>> > > You use the ADMBase variables, right?
>>> > My ADMBaseMcLachlanTester thorn (
>>> math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this
>>> issue, and uses *only* ADMBase::alp as input. I would encourage everyone
>>> interested to take a look at this thorn, as
>>> > 1) there's nothing complicated about the ADMBaseMcLachlanTester thorn
>>> at all (67 lines of code, including ccl files, includes, and whitespace)
>>> > 2) the included parfiles run on a single desktop computer needing only
>>> 5GB of RAM
>>> > 3) it takes only ~15 mins to evolve to timestep 192
>>> >
>>> > > Could you output the variables your analysis depends on, and see if
>>> the nan values come from there?
>>> > Absolutely! I just performed a run with ADMBaseMcLachlanTester,
>>> setting myadmbaselapse to alp every 64 iterations, and confirmed that there
>>> are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp
>>> (at any iteration, upto and including iteration 192), while again, there
>>> are a very large number of undefined values in myadmbaselapse at iteration
>>> 192...
>>> >
>>> > So myadmbaselapse is not undefined because ADMBase::alp is undefined.
>>>
>>> That tells you that alp is defined at the point it is output, but it
>>> doesn't tell you what its state was at the time myadmbaselapse used it.  I
>>> think you can add a call to CCTK_OutputVarAs or something in the routine,
>>> and you will get an output at that point.
>>> http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2.html#x4-139000A2
>>>
>>> So maybe something like
>>>
>>> CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse",
>>> "myadmbaselapse_when_set")
>>>
>>> and you should get your 2D output at the point the variable is set.  You
>>> could do the same for alp, to see what it is being set from.  Or you could
>>> use printf.
>>>
>>> --
>>> Ian Hinder
>>> http://members.aei.mpg.de/ianhin
>>>
>>>
>>
>> --
>> Ian Hinder
>> http://members.aei.mpg.de/ianhin
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20150602/e7015890/attachment-0001.html