Dear all,
I am writing to try to clarify a few issues that I've been facing regarding the schedule options "global loop-local" and "local" for an analysis thorn.
I have a thorn with a group scheduled as
schedule GROUP UAv_Analysis_Group at ANALYSIS after AHFinderDirect_maybe_do_masks
In this group, a function is dedicated to computing some grid functions:
schedule UAv_Analysis_gfs in UAv_Analysis_Group { LANG: Fortran SYNC: dE_gf_volume SYNC: dJ_gf_volume SYNC: quadrupole_gf_volume SYNC: density_rho SYNC: density_p } "Calculate grid functions"
I encounter some issue (described below) when using, in this schedule block, the option
OPTIONS: global loop-local
instead of the default "local". I was wondering about the "global loop-local" option because, in thorn ADMMass, it seems like the way to go (the schedule.ccl file contains comments about it). Using "local" yields the expected behavior, at least in the configurations which I have experimented with. Since I don't have a fine understanding of these options and how they affect the schedule, I'm willing to keep "local", and yearn to understand how to choose properly.
The problem that I'm facing with "global loop-local" is the following (all other things identical). I'm using a parameter file with Carpet, having 1 center and some refinement levels. My UAv_Analysis_gfs function contains an initial if statement that returns directly if cctk_iteration is not a multiple of a parameter do_analysis_every. The function sets a grid function called density_rho.
At the initial time, only the finest refinement level has initialized values. For later iterations, if do_analysis_every = 2^N, then for the output of density_rho, the finest level is initialized, the next N levels are uninitialized, and the remaining levels are initialized.
For instance, with CarpetRegrid2::num_levels_1 = 9 and do_analysis_every = 8, level 8 has values, levels 7, 6 and 5 don't, and levels 4 to 0 have values (except at iteration 0). In the extreme case do_analysis_every = 1, all levels have values. In the extreme case do_analysis_every = 256 (i.e every coarse level time step), only level 8 has values.
These issues do not occur with option "local".
Thank you for the insight and explanations that you can provide. I'm at your disposal for more information or details that could be useful.
Sincerely,
Jordan Nicoules