Sorry for the late reply, was (and am) still testing around...
Here is the schedule tree for my thorn:
disk_analysis::Disk_Analysis_Setup_LoopCounter: [global]
while (Disk_Analysis::nradii_LoopCounter)
GROUP DISK_ANALYSE: Disk_analysis main loop
disk_analysis::Disk_Analysis_Decrement_LoopCounter: [global]
disk_analysis::Disk_Analysis_calculate_J_gridpoint_shell: [global] [loop-local]
disk_analysis::Disk_Analysis_reduce_J_total_shell: [global]
disk_analysis::Disk_Analysis_calculate_angles_shell: [global]
end while
disk_analysis::Disk_Analysis_extraction_plane: [global]
GROUP DISK_GLOBAL_QUANTITIES:
disk_analysis::Disk_Analysis_set_global_quantities: [global] [loop-local]
disk_analysis::Disk_Analysis_reduce_global_quantities: [global]
disk_analysis::Disk_Analysis_set_global_specific_quantities: [global] [loop-local]
The problem is the following:
I want to calculate some quantities in the function disk_analysis::Disk_Analysis_set_global_specific_quantities, these will be grid functions that I want to output (in 1D and 3D)
however, the calculation depends on disk_analysis::Disk_Analysis_extraction_plane, where I decide if I need to rotate the coordinate system in which I later want to calculate the grid functions... it basically calculates and stores a rotation matrix, as well as an integer switch that later tells us if we need to rotate..
now, when scheduling disk_analysis::Disk_Analysis_set_global_specific_quantities as global loop-local, the values obtained in disk_analysis::Disk_Analysis_extraction_plane (which i store as SCALARS and have assigned storage for) are the correct ones, however, I cannot output the grid functions calculated, which I guess is due to this response from Roland in the mailing list earlier this year:
On Tue, Jun 2, 2015 at 1:46 PM, Roland Haas <rhaas at aei.mpg.de> wrote:
> Hello all,
>
> > If you are scheduling everything in GLOBAL (which is the same as
> > GLOBAL-LATE in ANALYSIS) then your OUTPUT will be incorrect since OUTPUT
> > is happening in LEVEL mode thus refinement level 3 is output before the
> > GLOBAL routine (which runs along with level 0) is executed.
> Oha, obvious not the right statement. GLOBAL-LATE will happen last but
> this happens to be the highest refinement level. Still the effect is the
> same OUTPUT happens before GLOBAL-LATE for all but one refinement level
> (that refinement level being the finest one).
>
> This can be seen in Carpet's Evolve.cc file in the CallAnalysis routine.
>
> Yours,
> Roland
now, if i want correct output, i can use either local mode or global-early loop-local mode..
then however, the values of the rotation matrix and the integer switch of actually using it are not correct, because the routine seems to be called now not in the order indicated by the schedule tree, but rather before the values have been set...
so: when using global loop-local in disk_analysis::Disk_Analysis_set_global_specific_quantities, the grid functions are not output correctly (only in the innermost refinement level)
and when using local or global-early loop-local the function disk_analysis::Disk_Analysis_set_global_specific_quantities cannot access the data calculated by disk_analysis::Disk_Analysis_extraction_plane..
is there an obvious way around this? ie getting the correct scheduling for both being able to use the values in the function AND outputting the calculated grid functions correctly??
best wishes,
Vassili