Hi,
If I add a thorn to a simulation part-way though, its Cactus variables won't be present in the checkpoint file on the first recovery. The code which detects this is in Carpet/CarpetIOHDF5/src/Input.cc and the warning is:
CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, "variable '%s' timelevel %d has not been read", fullname, tl);
In this case, there is no final error message and abort, unlike in the case of partially-read gridfunctions. Is this a deliberate decision, because for partially-read gridfunctions the results are definitely wrong, whereas not reading variables at all might be OK? In my particular case, the variables are scalar integers. What value would the variables have in this case? Does Carpet initialise variables to 0? I seem to see 0, but haven't checked carefully.
Hello Ian,
In this case, there is no final error message and abort, unlike in the case of partially-read gridfunctions. Is this a deliberate decision,
Yes, this is deliberate to allow thorns to be activated in ongoing simulations. There is a level 2 warning about the non-read in variables so it should not go unnoticed (in theory at least).
because for partially-read gridfunctions the results are definitely wrong, whereas not reading variables at all might be OK? In my particular case, the variables are scalar integers. What value would the variables have in this case? Does Carpet initialise variables to 0? I seem to see 0, but haven't checked carefully.
They would be whatever Carpet or the memory allocator initialized them to. CCTK_Initial does not run (normally) during checkpoint recovery. You can have CCTK_Initial run *before* checkpoint recovery (with values in the recoverd variables overwritten by the recovered values) by setting Cactus::recovery_mode to "relaxed" (see flesh/src/param.ccl). This may be what you had in mind.
Yours, Roland
On 6 Oct 2016, at 14:35, Roland Haas rhaas@illinois.edu wrote:
Hello Ian,
In this case, there is no final error message and abort, unlike in the case of partially-read gridfunctions. Is this a deliberate decision,
Yes, this is deliberate to allow thorns to be activated in ongoing simulations. There is a level 2 warning about the non-read in variables so it should not go unnoticed (in theory at least).
because for partially-read gridfunctions the results are definitely wrong, whereas not reading variables at all might be OK? In my particular case, the variables are scalar integers. What value would the variables have in this case? Does Carpet initialise variables to 0? I seem to see 0, but haven't checked carefully.
They would be whatever Carpet or the memory allocator initialized them to. CCTK_Initial does not run (normally) during checkpoint recovery. You can have CCTK_Initial run *before* checkpoint recovery (with values in the recoverd variables overwritten by the recovered values) by setting Cactus::recovery_mode to "relaxed" (see flesh/src/param.ccl). This may be what you had in mind.
Aha, that's interesting. Though I guess it would probably confuse a lot of thorns, so I'm not sure how useful it in in practice. The initial data solver would run again, for example. I would have expected Carpet to initialise variables to poison, so I should be seeing poison. What I am doing now is initialising the thorn in basegrid, and checking an "initialised" Cactus variable. If this is 1, then I don't do anything. If it's anything other than 1 (poison, zero, etc) then I assume the thorn has never been initialised. This is probably not an approved, elegant solution, but I expect it will work.
-- Ian Hinder http://members.aei.mpg.de/ianhin
Hello Ian,
Aha, that's interesting. Though I guess it would probably confuse a lot of thorns, so I'm not sure how useful it in in practice. The initial data solver would run again, for example.
Correct. The current default is "safer" it seems. The "relaxed" setting I have never seen used myself.
I would have expected Carpet to initialise variables to poison, so I should be seeing poison.
That is also what I would have expected. Have you set both posion_new_memory and poison_new_timelevels turned on?
What I am doing now is initialising the thorn in basegrid, and checking an "initialised" Cactus variable. If this is 1, then I don't do anything. If it's anything other than 1 (poison, zero, etc) then I assume the thorn has never been initialised. This is probably not an approved, elegant solution, but I expect it will work.
Well it's kind of what you have to do. You should be able to just initialize in basegrid all the time, since the basegrid values will be overwritten during checkpoint recovery as far as I remember.
Yours, Roland
On 6 Oct 2016, at 17:39, Roland Haas rhaas@illinois.edu wrote:
Hello Ian,
Aha, that's interesting. Though I guess it would probably confuse a lot of thorns, so I'm not sure how useful it in in practice. The initial data solver would run again, for example.
Correct. The current default is "safer" it seems. The "relaxed" setting I have never seen used myself.
I would have expected Carpet to initialise variables to poison, so I should be seeing poison.
That is also what I would have expected. Have you set both posion_new_memory and poison_new_timelevels turned on?
Hmm. I thought that I had, but now I see in my parameter file:
Carpet::poison_new_timelevels = no Carpet::check_for_poison = no
which is a bit surprising. This might come from the old issue of the intersection between outer boundaries and interpatch boundaries not being correctly initialised by the combination of Interpolate2 and McLachlan. Last time I checked though, this may have been fixed. If you don't use poison, does Carpet zero the variables, or leave them uninitialised? In which case, does the OS zero them? Presumably new memory from the OS is zeroed (for security/privacy reasons), but if the memory has been malloc/freed already by the cactus process, then maybe it is random.
What I am doing now is initialising the thorn in basegrid, and checking an "initialised" Cactus variable. If this is 1, then I don't do anything. If it's anything other than 1 (poison, zero, etc) then I assume the thorn has never been initialised. This is probably not an approved, elegant solution, but I expect it will work.
Well it's kind of what you have to do. You should be able to just initialize in basegrid all the time, since the basegrid values will be overwritten during checkpoint recovery as far as I remember.
Aha! Interesting. Good. So I can just move the initialisation to basegrid and remove my 'initialised' flag. This way, the variables will be initialised, and then only overwritten from the checkpoint file if the thorn was active in the previous segment.
Now, another related question. Is it possible to change the number of CarpetRegrid2 centres during evolution? The num_centres parameter is non-steerable, and indeed CarpetRegrid2 only looks at the centre parameters initially (otherwise every recovery would overwrite the dynamical centre variables with the parameter values). Ideally, I would have set num_centres to a larger value when I started the run, but I didn't.
We just need to make sure that CarpetRegrid2 runs INIT_CENTRE (from initialise.cc) on any newly added centres on recovery. This is done in WRAGH, and then similar to the previous issue, overwritten by recovery. So this should just work?
-- Ian Hinder http://members.aei.mpg.de/ianhin
Hello Ian,
ah, these are all tricky questions.
which is a bit surprising. This might come from the old issue of the intersection between outer boundaries and interpatch boundaries not being correctly initialised by the combination of Interpolate2 and McLachlan. Last time I checked though, this may have been fixed. If you don't use poison, does Carpet zero the variables, or leave them uninitialised? In which case, does the OS zero them? Presumably new memory from the OS is zeroed (for security/privacy reasons), but if the memory has been malloc/freed already by the cactus process, then maybe it is random.
Yes, the OS would zero, malloc does not have to. new() on the other hand will also zero (you'd have to check if CarpetLib uses new() or malloc() or posix_memalign() to get new memory, I believe it is actually malloc).
Now, another related question. Is it possible to change the number of CarpetRegrid2 centres during evolution? The num_centres parameter is non-steerable, and indeed CarpetRegrid2 only looks at the centre parameters initially (otherwise every recovery would overwrite the dynamical centre variables with the parameter values). Ideally, I would have set num_centres to a larger value when I started the run, but I didn't.
We just need to make sure that CarpetRegrid2 runs INIT_CENTRE (from initialise.cc) on any newly added centres on recovery. This is done in WRAGH, and then similar to the previous issue, overwritten by recovery. So this should just work?
Yes that would work. You just have to make sure you can identify new levels, eg. by their number of levels being -42 or similar. The alternative is to use the Trigger thorn and set the grid scalars (see the NSNS gallery parfile and look for
Trigger::Trigger_Steered_Scalar[1] = "CarpetRegrid2::num_levels[2]"
Since you can change the grid scalars and "activate" more levels.
Yours, Roland
Ian
The operating system will zero all pages before the application sees them. Malloc and friends will reuse memory from the operating system without zeroing it. In practice, the first malloc calls will see all zeros, later ones (especially for smaller regions) might see non-zeros. Carpet does not set anything to zero, but it can set things to nan if you ask it to. MoL will also set the RHS to zero by default before every iteration.
-erik
On Fri, Oct 7, 2016 at 9:31 AM, Roland Haas rhaas@illinois.edu wrote:
Hello Ian,
ah, these are all tricky questions.
which is a bit surprising. This might come from the old issue of the intersection between outer boundaries and interpatch boundaries not being correctly initialised by the combination of Interpolate2 and McLachlan. Last time I checked though, this may have been fixed. If you don't use poison, does Carpet zero the variables, or leave them uninitialised? In which case, does the OS zero them? Presumably new memory from the OS is zeroed (for security/privacy reasons), but if the memory has been malloc/freed already by the cactus process, then maybe it is random.
Yes, the OS would zero, malloc does not have to. new() on the other hand will also zero (you'd have to check if CarpetLib uses new() or malloc() or posix_memalign() to get new memory, I believe it is actually malloc).
Now, another related question. Is it possible to change the number of CarpetRegrid2 centres during evolution? The num_centres parameter is non-steerable, and indeed CarpetRegrid2 only looks at the centre parameters initially (otherwise every recovery would overwrite the dynamical centre variables with the parameter values). Ideally, I would have set num_centres to a larger value when I started the run, but I didn't.
We just need to make sure that CarpetRegrid2 runs INIT_CENTRE (from initialise.cc) on any newly added centres on recovery. This is done in WRAGH, and then similar to the previous issue, overwritten by recovery. So this should just work?
Yes that would work. You just have to make sure you can identify new levels, eg. by their number of levels being -42 or similar. The alternative is to use the Trigger thorn and set the grid scalars (see the NSNS gallery parfile and look for
Trigger::Trigger_Steered_Scalar[1] = "CarpetRegrid2::num_levels[2]"
Since you can change the grid scalars and "activate" more levels.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Ian Hinder -
Roland Haas