[Users] Checkpoint/recovery inconsistencies

[Erik Schnetter]

On Mon, Aug 1, 2011 at 10:37 AM, Roland Haas
<roland.haas at physics.gatech.edu> wrote:
> Hello Erik, all,
>
>> After a bit of clean-up, this is the final patch I intend to commit.
>> In addition to what was discussed earlier, this also introduces a new
>> MoL schedule group MoL_PostStepModify:
>>
>> Phyics thorns can apply enforce constraints in PostStepModify. The
>> difference between PostStep and PostStepModify is that PostStep is
>> scheduled at many other occasions, whereas the PostStepModify is only
>> scheduled during evolution.
>
> So PostStepModify should contain eg. evolution boundary conditions and
> dissipation (eg.) while PostStep would contain symmetry boundary conditions,
> con2prim etc? Would that also mean that routines scheduled in PostStep are
> supposed to be idempotent (ie. applying them twice does not change the
> values on the grid)? Note that the later excludes the current implementation
> of Con2Prim (I think) since they will do at least grhydro_count_min Newton
> iterations (and therefore always change values).

Almost: MoL_PostStepModify would contain those evolution steps that
are independent of MoL. Dissipation is applied to the RHS, so it
doesn't go into this group. Currently, only enforcing the BSSN
constraints is scheduled here.

Yes, MoL_PostStep routines are supposed to be idempotent, since they
are applied "randomly" at various occations, e.g. after regridding. If
there is a regridding step that doesn't modify level L, then
MoL_PostStep may still be executed on level L, and if a routine is not
idempotent, the results will differ. I was thinking of con2prim
myself. In a production run, with MPI and OpenMP and compiler
optimisations, things will differ anyway "randomly", but it may be
nice to have a certain operating mode for con2prim which makes it
idempotent, maybe only for a simple EOS.

> Otherwise I like the idea of separating enforcement of constraints from
> evolution-type calculations.
>
> I have a comment for this part of the patch:
>
> --8<--
> Index: arrangements/EinsteinEvolve/GRHydro/schedule.ccl
> ===================================================================
> --- arrangements/EinsteinEvolve/GRHydro/schedule.ccl    (revision 251)
> +++ arrangements/EinsteinEvolve/GRHydro/schedule.ccl    (working copy)
> @@ -983,9 +983,9 @@
>   } "Reset the atmosphere"
>  }
>
> -schedule group HydroBase_Boundaries IN MoL_Evolution AFTER MoL_Step
> -{
> -} "HydroBase Boundary conditions group"
> +#schedule group HydroBase_Boundaries IN MoL_Evolution AFTER MoL_Step
> +#{
> +#} "HydroBase Boundary conditions group"
> --8<--
>
> the reason why this extra Boundary group was scheduled was that
> ResetAtmoshphere runs in MoL_Evolution AFTER MoL_Step. ResetAtmoshpere uses
> values in space_mask to reset some points, however space_mask was only set
> for the interior points (since it depends in part on the RHS) so in order to
> have consistent data one has to SYNC and apply boundary conditions (though
> con2prim is not required I think). Is this still true?

Thanks. I added HydroBase_Boundaries at this location again. Note that
this group may sync and hence may be expensive -- skipping this group
unless necessary may be a good idea.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/