[Users] problem in analysis thorn

[Haas, Roland]

Hello Fabrizio,

I am not quite sure what may be happening, but here's my guess:

options: global loop-local

will call your scheduled function once on each refinement level (and
component) at every since iteration. This includes calling on reflevel
0 at iteration 1 (at which point an normal "local" scheduled routine
would not be called for that refinement level). 

To add to the difficulties, if you look at
repos/carpet/Carpet/src/Evolve.cc:, then you an see that global routines will run *first* ie along with the first "local" routines. This means that if the global routine accesses data that is only computed by "local" routines then this data will not be available (on all refinement levels) when the global routine runs. You may see better results by using "global-late" instead of "global".

Carpet has an option Carpet::veryverbose = yes which will print out
exactly when each scheduled function is called (much more details than
just the schedule printout).

Note that this is mostly a blind guess. You could try and see what
would happen if eg you have only a single refinement level (in which
case I would expect the issue to go away if my guess is correct).

I would also try and move the routines out of MoL_PostStep and
AddToTmunu and schedule them explicitly in EVOL (or in a group that is
explicitly in EVOL).

Yours,
Roland

> Hi all.
> 
> I'm having a problem, probably in the scheduling, but I'm not able to
> understand why this is happening. I attached two files .txt where I report
> the schedule.ccl file of my thorn and the grid parameters in my parfile.
> What happen is the following; I have these variables defined in the
> interface.ccl like this:
> 
> CCTK_REAL masscenter_point type = GF Timelevels = 3
> tags='Prolongation="none" tensortypealias="Scalar" '
> {
> 
>   masscenterX_point
>   masscenterY_point
> 
> }"mass center to be used in the global reductions"
> 
> 
> CCTK_REAL global_masscenter type = SCALAR
> {
> 
>  masscenterX
>  masscenterY
> 
> }"mass center coordinates"
> 
> 
> What I want to do is evaluate in every grid point the values for
> masscenterX_point and masscenterY_point (I do this in the function
> my_analysis_point_calc) and then integrate them (using CCTK_Reduce) over
> the entire domain and save the result in the scalar variables masscenterX
> and masscenterY (and I do it in the function my_analysis_reduce). So far
> everything seems to work well, if I plot the masscenter it has reasonable
> values. The problem happens when, in the same thorn, I want to use the
> values of masscenterX and masscenterY in another function (
> my_analysis_modes_point_calc) to evaluate some other variable (let's call
> it mode_m1) point by point and then integrate it (in
> my_analysis_modes_reduce).
> 
> 
> At the initial time I get a reasonable value for mode_m1, but the second
> time I print it ( after 12 timesteps) I get nan, the third time I print it
> (after 24 timesteps) I get a reasonable value and after that it seems to be
> fine for the rest of the simulation. When I print masscenterX and
> masscenterY they have reasonable values at every iteration.  This is the
> output:
> "
> 
> #1:iteration 2:time 3:data
> #data columns: 3:masscenterX 4:masscenterY
> 0 0 2.04233950307922e-07 1.43009709532618e-07
> 12 0.6 1.64952865679123e-07 1.15302233769628e-07
> 24 1.2 2.59441645250822e-08 1.8122269479532e-08
> 
> #1:iteration 2:time 3:data
> #data columns: 3:mode_m1
> 0 0 5.09771568069084e-09
> 12 0.6 -nan
> 24 1.2 5.98971400034287e-10
> 
> "
> 
> I thought that the problematic line in my code could be this one:
> 
>             phi = atan((yy-*masscenterY)/(xx-*masscenterX));
> 
>             if(((xx-*masscenterX)<0.0)){
>                 phi = atan((yy-*masscenterY)/(xx-*masscenterX)) +
> 3.14159265358979323846;
>             }
> 
> where I try to use the values of masscenterX and masscenterY inside the
> function "my_analysis_modes_point_calc". If I change that part not using
> masscenterX and masscenterY I don't get nans anymore in mode_m1.
> 
> After doing different tests, I actually found out that if I try to plot
> scalar output every 4 iterations I get also NaN in masscenter at time 0.2;
> this is the output in this case:
> "
> #1:iteration 2:time 3:data
> #data columns: 3:masscenterX 4:masscenterY
> 
> 0 0 2.04233950307922e-07 1.43009709532618e-07
> 4 0.2 -nan -nan
> 8 0.4 1.76472702563281e-07 1.23428120452789e-07
> 12 0.6 1.64952865679123e-07 1.15302233769628e-07
> 16 0.8 1.05057935424561e-07 7.33797284607643e-08
> 20 1 5.6677119219615e-08 3.93971526043537e-08
> 24 1.2 2.59441645250822e-08 1.8122269479532e-08
> 28 1.4 5.57004919317397e-08 3.90090013497981e-08
> 32 1.6 1.2454179781868e-08 8.57409404496548e-09
> 36 1.8 7.81397952778142e-08 5.4397631432516e-08
> 
> #1:iteration 2:time 3:data
> #data columns: 3:mode_m1
> 
> 0 0 5.09771568069084e-09
> 4 0.2 -nan
> 8 0.4 -nan
> 12 0.6 -nan
> 16 0.8 2.49105795013804e-09
> 20 1 -nan
> 24 1.2 5.98971400034287e-10
> 28 1.4 3.14686031011036e-10
> 32 1.6 2.57981958462494e-10
> 36 1.8 6.79137649228778e-10
> 
> "
> 
> I'm not really sure what is going on here, so I hope somebody can explain
> me where is the mistake. Thank you.
> 
> Best,
> Fabrizio



-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20191004/589c65f8/attachment-0001.bin