Hello Gabriele,
I seem to have trouble sending email to the ET mailing list server using my regular UIUC email address, so maybe using GMail will work better.
Yours, Roland
Begin forwarded message:
Date: Sat, 1 Jan 2022 10:36:00 -0500 From: Roland Haas rhaas@illinois.edu To: Gabriele Bozzola bozzola.gabriele@gmail.com Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] PreSync and recovery from checkpoint
Hello Gabriele,
I have a fork of ProcaEvolve ( https://urldefense.com/v3/__https://bitbucket.org/Sbozzolo/proca/src/master/... ) where I added the READS/WRITES statements for PreSync and other features. One of such features is computing the constraint, but I found out that there is some problem when PreSync is running with `mixed-error`.
hmm.
I have a write statement for chi in CCTK_INITIAL, which is not called when restarting a simulation. Nonetheless, chi should be available everywhere because it is read from a checkpoint. What is the correct thing to do in this scenario?
When grid functions are read in from a checkpoint then they are marked as being valid everywhere. This is done in CarpetIOHDF5/src/Input.cc in the Recover function in the block that starts with
// check that all variables have been read completely on this mglevel/reflevel
specifically in line 793 of that file:
gf->set_valid(mglevel, reflevel, tl, CCTK_VALID_EVERYWHERE);
If these variables are not checkpointed (likely, given that they are only constraints and thus not usually considered "important" to checkpoint) then you would have to add code to say POST_RECOVER_VARIABLES to initialize them (eg to 0). This is a good idea to do even without considering PreSync since it avoids having random junk data (or poison) in the constraint output for a couple of steps (depending on mesh refinement details) after recovery.
Something like:
schedule Proca_ClearConstraints AT POST_RECOVER_VARIABLES { LANG: Fortran } "Initialize constraint values after recovery"
You can also try to to Proca_Constraints in post-recover variables but, since the constraints require derivatives and those cannot be (correctly) computed in post-recover-variables, the constraints for the checkpointed timestep will (possibly I would have to think about exactly how checkpointing, prolongation and time stepping interact) differ before and after recovery.
Also POST_RECOVER_VARIABLES only runs the scheduled function on timelevel 0 (ie current time) and not any past timelevels so if there are multiple time levels to the constraint variables, only tl=0 will be initialized.
Yours, Roland
Hi Roland,
Thanks for your response, which made me realize that the conformal factor is not checkpointed, so the error makes perfect sense.
Also POST_RECOVER_VARIABLES only runs the scheduled function on
timelevel 0 (ie current time) and not any past timelevels so if there are multiple time levels to the constraint variables, only tl=0 will be initialized.
In this specific case, I can either turn on checkpointing for the BSSN variables in ProcaEvolve, or I can recompute them from the ADM ones after recovery. What is the correct way to re-initialize a derived variable after recovery?
Thanks, Gabriele
On Mon, Jan 3, 2022 at 2:13 PM Roland Haas rhaas80@gmail.com wrote:
Hello Gabriele,
I seem to have trouble sending email to the ET mailing list server using my regular UIUC email address, so maybe using GMail will work better.
Yours, Roland
Begin forwarded message:
Date: Sat, 1 Jan 2022 10:36:00 -0500 From: Roland Haas rhaas@illinois.edu To: Gabriele Bozzola bozzola.gabriele@gmail.com Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] PreSync and recovery from checkpoint
Hello Gabriele,
I have a fork of ProcaEvolve (
https://urldefense.com/v3/__https://bitbucket.org/Sbozzolo/proca/src/master/... )
where I added the READS/WRITES statements for PreSync and other features. One of such features is computing the constraint, but I found out that there is some problem when PreSync is running with `mixed-error`.
hmm.
I have a write statement for chi in CCTK_INITIAL, which is not called when restarting a simulation. Nonetheless, chi should be available everywhere because it is read from a checkpoint. What is the correct thing to do in this scenario?
When grid functions are read in from a checkpoint then they are marked as being valid everywhere. This is done in CarpetIOHDF5/src/Input.cc in the Recover function in the block that starts with
// check that all variables have been read completely on this mglevel/reflevel
specifically in line 793 of that file:
gf->set_valid(mglevel, reflevel, tl, CCTK_VALID_EVERYWHERE);
If these variables are not checkpointed (likely, given that they are only constraints and thus not usually considered "important" to checkpoint) then you would have to add code to say POST_RECOVER_VARIABLES to initialize them (eg to 0). This is a good idea to do even without considering PreSync since it avoids having random junk data (or poison) in the constraint output for a couple of steps (depending on mesh refinement details) after recovery.
Something like:
schedule Proca_ClearConstraints AT POST_RECOVER_VARIABLES { LANG: Fortran } "Initialize constraint values after recovery"
You can also try to to Proca_Constraints in post-recover variables but, since the constraints require derivatives and those cannot be (correctly) computed in post-recover-variables, the constraints for the checkpointed timestep will (possibly I would have to think about exactly how checkpointing, prolongation and time stepping interact) differ before and after recovery.
Also POST_RECOVER_VARIABLES only runs the scheduled function on timelevel 0 (ie current time) and not any past timelevels so if there are multiple time levels to the constraint variables, only tl=0 will be initialized.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Hello Gabriele,
Sorry for the delay. Your email was flagged as "spam" by the University of Illinois and I forgot to respond after noting that your email was an "unanswered question" week before last week.
Steve responded to you as well in the meantime I believe.
In this specific case, I can either turn on checkpointing for the BSSN variables in ProcaEvolve, or I can recompute them from the ADM ones after recovery. What is the correct way to re-initialize a derived variable after recovery?
If you want to be consistent with eg ML_BSSN then checkpointing and recovering is the way to go.
Recomputing would reduce the checkpoint size a bit. Usually that is not a problem (speed or size wise). One has to be *very* careful though that one can exactly (to the last bit) recompute the variable though (eg there must not be anything that runs after BSSNtoADM that modifies the BSSN variables [eg boundary routines, constraint projections, whatnot]). If one cannot guarantee that, then recovering from checkpoints (and not touching the recovered variables) would be preferable. For historic reasons there is (at least) one thorn in ET (GRHydro) that violates this, namely it runs con2prim in post-recover-variables (since it runs in mol_poststep and mol_poststep runs in that bin) but con2prim is not going to produce the same results due to its result depending on the initial guess which is the values of the primitive variables in hydrobase ("previous" values during evolution, "just recovered values" during recovery). This behavious occasionally causes headaches when a run fails / succeeds depending on which checkpoint one recovers from.
Yours, Roland
users@lists.einsteintoolkit.org
-
Gabriele Bozzola -
Roland Haas -
Roland Haas