[Users] Schedule options and uninitialized refinement levels

[Roland Haas]

Hello Jordan,

> instead of the default "local". I was wondering about the "global
> loop-local" option because, in thorn ADMMass, it seems like the way
> to go (the schedule.ccl file contains comments about it). Using
> "local" yields the expected behavior, at least in the configurations
> which I have experimented with. Since I don't have a fine
> understanding of these options and how they affect the schedule, I'm
> willing to keep "local", and yearn to understand how to choose
> properly.

Short answer: it's (mostly) only the output that is wrong (if you were
to look at the data on the grid chances are it would look correct). But
the data will never be exactly correct unless you compute things in
EVOL.

Very long answer:

Scheduling with AMR and subcycling in time is fairly complicated. 

As a general rule (there are very few exemptions, and ANALYSIS is not
one of time):

* for each Cactus schedule bin (INITIAL, EVOL, POSTSTEP, ANALYSIS plus
  some others) the full schedule is executed for each iteration and each
  refinement level

* coarse refinement levels are executed *first* with finer refinement
  levels next

* scheduled functions with "OPTION: global" are executed exactly once
  for each iteration, either along with the coarsest refinement level
  for this iteration (early, forced by using global-early instead of
  global) or with the finest level (late, forced by using global-late
  instead of global) and skipped in the other traversals of the
  schedule, In ANALYSIS and POSTSTEP "global" is equivalent to
  "global-late" ie a scheduled routine marked as "global" will execute
  along with the "local" scheduled routines on the finest level.

* "loop-local" means that there will be an explicit loop over the
  refinement levels and the scheduled function will be called on each
  refinement level (and each grid component), in a similar manner to a
  "local" routine (but this will all happen at the same time as the
  *single* refinement level that "global" attaches to executes its
  scheduled functions)

* there's one more subtlety involved, namely there's a difference
  between EVOL and ANALYSIS concerning which refinement levels are
  considered eligible for execution. Basically speaking in EVOL a
  refinement level executes its scheduled grid functions if it is
  possible to evaluate the RHS of the equation of motion, while in
  ANALYSIS a grid function executes once evolution for that time has
  finished. In practise this means that for iteration=1 one evaluates
  the RHS for refinement level 0, 1, 2, ..., N since they all can
  evaluate the RHS (and the result is actually needed for boundary
  conditions for the finer levels), while *only* the finest level,
  refinement level N will have finished its evolution step at iteration
  1 some only it will run in ANALYSIS.

So what ADMMass does (see the comments in there) is having to ensure
some fixed ordering between functions that are global and those that
need to access grid functions.

Specifically there is ADMMass_SetLoopCounter which is "global", so
executes along with the *last* refinement level. Without the "global
loop-local" the ADMMass_Surface scheduled function would run for each
refinement level and thus would run on levels 0, 1, 2... before
ADMMass_SetLoopCounter (which is "global" so runs at the same time as
the finest level) has had a chance of running. 

It's complicated.

> The problem that I'm facing with "global loop-local" is the following
> (all other things identical). I'm using a parameter file with Carpet,
> having 1 center and some refinement levels. My UAv_Analysis_gfs
> function contains an initial if statement that returns directly if
> cctk_iteration is not a multiple of a parameter do_analysis_every.
> The function sets a grid function called density_rho.
> 
> At the initial time, only the finest refinement level has initialized
> values. For later iterations, if do_analysis_every = 2^N, then for
> the output of density_rho, the finest level is initialized, the next
> N levels are uninitialized, and the remaining levels are initialized.
> 
> For instance, with CarpetRegrid2::num_levels_1 = 9 and
> do_analysis_every = 8, level 8 has values, levels 7, 6 and 5 don't,
> and levels 4 to 0 have values (except at iteration 0). In the extreme
> case do_analysis_every = 1, all levels have values. In the extreme
> case do_analysis_every = 256 (i.e every coarse level time step), only
> level 8 has values.

Hmm, this should compute the correct values for most of the grid points.
However my suspicion is that when you say that is does not contain the
expected values then that is based on what you see in output files if
you add some sort of `out_var = "dE_gf_volume"` to the parameter files,
yes?

In that case you will indeed see bad data in output. Namely looking at
the description above notice that "global" in ANALYSYS is "global-late"
i.e. executes with the last refinement level (the finest one). However
*output* is also done level by level, that is output for refinement
level 0 is done first, along with all the scheduled functions for
refinement level 0. So since your calculation with "global
loop-local" runs *last*, things are not yet computed and you see old or
garbage data in the output files.

> These issues do not occur with option "local".

Correct (well at least nothing obviously wrong is output), in this case
the data is computed just before it is output (but continue reading for
some subtle errors).

*If* your calculations are all pointwise (not derivatives etc.) and
you have only a *single* time level active for the grid functions that
you compute *then* (and only then) can you compute things in ANALYSIS.

If your do_analysis_every variable is set such that things are *only*
computed (and output, or otherwise used) when all refinement levels are
in sync (ie every_coarse) then you can do a SYNC (but you still don't
need more than 1 timelevel) without pulling in bad data. However your
answer in parts of the grid will be wrong since there will not be any
restriction of the data computed on a fine grid to the coarse grid
(which happens only when going from EVOL to POSTSTEP). This matters
only for stencil operations but not for operations that are
pointwise local (since they'd compute the same value either way). This
will (mostly...) not effect any interpolation results and will not (I
think) affect reductions (min, max, norm2 whatnot). But ... this is very
tricky.

The safest strategy is to compute everything in EVOL (not POSTSTEP), in
particular if it involves stencil operations (in which case it is
almost a requirement). You will need 3 timelevels though, at least if
you plan on doing interpolations or reduction output at times other than
every_coarse (and you need to compute at every iteration, or your are
on very very thin ice).

Your schedule statement makes me suspect that your operation is not
fully pointwise and does instead involve a stencil operation. In that
case, the best you can achieve is making sure that there is only 1 time
level for the variables and that you compute only when all refinement
levels are in sync in time (so every coarse). Note that you will still
get incorrect results in the regions of the coarse grid that are
overlaid by the fine grid (no restriction so you are left with the
result computed on the coarse grid), but that may not matter much to
you.

> Thank you for the insight and explanations that you can provide. I'm
> at your disposal for more information or details that could be useful.

If you could provide the actual schedule.ccl and param.ccl files this
would be helpful. Right now I am at least partially guessing.

Yours,
Roland

-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20240923/64ee409d/attachment-0001.sig>