Hi,
When running long simulations with checkpointing, it might sometimes be useful to change certain parameters, especially I/O (e.g. the frequency of outputting a certain variable). Is this possible in Cactus/Simfactory? I notice that upon recovery (in the next re-start) Simfactory reads the parameters from a checkpoint file, not from any of the parfiles dumped within the job directory.
Any suggestions/responses will be highly appreciated.
Thank you, Dumsani
On 30 Sep 2016, at 16:04, dumsani g14n8326@campus.ru.ac.za wrote:
Hi,
When running long simulations with checkpointing, it might sometimes be useful to change certain parameters, especially I/O (e.g. the frequency of outputting a certain variable). Is this possible in Cactus/Simfactory? I notice that upon recovery (in the next re-start) Simfactory reads the parameters from a checkpoint file, not from any of the parfiles dumped within the job directory.
Any suggestions/responses will be highly appreciated.
Hi Dumsani,
When you run the simulation again, you give it a new parameter file. It is this new parameter file which is checked for steered parameters; Cactus doesn't look at the output from a previous run. Many parameters can be steered on recovery in this manner, by changing them in the parameter file. Be careful when steering the out_every parameters; you almost certainly want out_criterion = divisor, not the default of "iteration". "iteration" outputs if the current iteration number is N greater than the last output iteration. "divisor" outputs if the current iteration is divisible by N.
On Fri, Sep 30, 2016 at 05:40:50PM +0200, Ian Hinder wrote:
When you run the simulation again, you give it a new parameter file. It is this new parameter file which is checked for steered parameters; Cactus doesn't look at the output from a previous run. Many parameters can be steered on recovery in this manner, by changing them in the parameter file. Be careful when steering the out_every parameters; you almost certainly want out_criterion = divisor, not the default of "iteration". "iteration" outputs if the current iteration number is N greater than the last output iteration. "divisor" outputs if the current iteration is divisible by N.
I assume Dumsani refers to simfactory. How do you change the parameter file that simfactory tells Cactus to use for a restart? I admit that in that case I usually go in and change the file in the output directory (simulations/name/...), but there must be a better way. I didn't have to do that so often, so I never bothers to find out myself.
Does simfactory actually observe a --parfile option on submission of a restart? It probably will not complain if you try, but does it actually do what you want?
Frank
On 30 Sep 2016, at 17:47, Frank Loeffler knarf@cct.lsu.edu wrote:
On Fri, Sep 30, 2016 at 05:40:50PM +0200, Ian Hinder wrote:
When you run the simulation again, you give it a new parameter file. It is this new parameter file which is checked for steered parameters; Cactus doesn't look at the output from a previous run. Many parameters can be steered on recovery in this manner, by changing them in the parameter file. Be careful when steering the out_every parameters; you almost certainly want out_criterion = divisor, not the default of "iteration". "iteration" outputs if the current iteration number is N greater than the last output iteration. "divisor" outputs if the current iteration is divisible by N.
I assume Dumsani refers to simfactory. How do you change the parameter file that simfactory tells Cactus to use for a restart? I admit that in that case I usually go in and change the file in the output directory (simulations/name/...), but there must be a better way. I didn't have to do that so often, so I never bothers to find out myself.
Does simfactory actually observe a --parfile option on submission of a restart? It probably will not complain if you try, but does it actually do what you want?
That I do not know. It would be nice if it did. What I usually do is change the top-level SIMFACTORY/par/*.par file. I don't think simfactory reads anything from restarts; it only reads from the top-level SIMFACTORY directory when creating a new restart.
-- Ian Hinder http://members.aei.mpg.de/ianhin
Hi Frank, Ian, Everyone
Yes, I meant doing this via Simfactory. But at the same time I acknowledge the fact that it could as well be doable via some Cactus parameter... On that note, we have tried changing the parameter in the top-level SIMFACTORY directory under the job directory. However, looking at the parfile in the new checkpoint directory, it still bore the old value. We even went as far as changing the parameter in all parfiles in the previous checkpoint subdirectory, in addition to changing it in the top-level SIMFACTORY directory.
In any case, if you also suggest that then I'm going to give this a try again over the weekend (or on Monday - our computing center is on a shut down for the weekend).
Thank you Dumsani
On 30/09/2016 17:57, Ian Hinder wrote:
On 30 Sep 2016, at 17:47, Frank Loeffler <knarf@cct.lsu.edu mailto:knarf@cct.lsu.edu> wrote:
On Fri, Sep 30, 2016 at 05:40:50PM +0200, Ian Hinder wrote:
When you run the simulation again, you give it a new parameter file. It is this new parameter file which is checked for steered parameters; Cactus doesn't look at the output from a previous run. Many parameters can be steered on recovery in this manner, by changing them in the parameter file. Be careful when steering the out_every parameters; you almost certainly want out_criterion = divisor, not the default of "iteration". "iteration" outputs if the current iteration number is N greater than the last output iteration. "divisor" outputs if the current iteration is divisible by N.
I assume Dumsani refers to simfactory. How do you change the parameter file that simfactory tells Cactus to use for a restart? I admit that in that case I usually go in and change the file in the output directory (simulations/name/...), but there must be a better way. I didn't have to do that so often, so I never bothers to find out myself.
Does simfactory actually observe a --parfile option on submission of a restart? It probably will not complain if you try, but does it actually do what you want?
That I do not know. It would be nice if it did. What I usually do is change the top-level SIMFACTORY/par/*.par file. I don't think simfactory reads anything from restarts; it only reads from the top-level SIMFACTORY directory when creating a new restart.
-- Ian Hinder http://members.aei.mpg.de/ianhin
On 30 Sep 2016, at 21:56, dumsani g14n8326@campus.ru.ac.za wrote:
Hi Frank, Ian, Everyone
Yes, I meant doing this via Simfactory. But at the same time I acknowledge the fact that it could as well be doable via some Cactus parameter... On that note, we have tried changing the parameter in the top-level SIMFACTORY directory under the job directory. However, looking at the parfile in the new checkpoint directory, it still bore the old value. We even went as far as changing the parameter in all parfiles in the previous checkpoint subdirectory, in addition to changing it in the top-level SIMFACTORY directory.
In any case, if you also suggest that then I'm going to give this a try again over the weekend (or on Monday - our computing center is on a shut down for the weekend).
Hi Dumsani,
The top-level simfactory directory I am talking about is:
mysim/ SIMFACTORY/ <------ This one par/ mysim.par output-0000/ SIMFACTORY output-0001/ SIMFACTORY/ ...
i.e. there are SIMFACTORY directories inside each restart, and there is one at the very top level of the simulation. If you change the parameter file in this one, it should be copied into the output-NNNN/SIMFACTORY/par directory when the new restart is created. That was my understanding of how it works, though I admit that I have not used SimFactory 2 for a while. Do other people use this mechanism and can confirm that this is how it works?
Hi Ian, Your suggestion worked! That is, changing the parameters in the top-level SIMFACTORY directory. I just I should mention it.
On 01/10/2016 11:59, Ian Hinder wrote:
On 30 Sep 2016, at 21:56, dumsani <g14n8326@campus.ru.ac.za mailto:g14n8326@campus.ru.ac.za> wrote:
Hi Frank, Ian, Everyone
Yes, I meant doing this via Simfactory. But at the same time I acknowledge the fact that it could as well be doable via some Cactus parameter... On that note, we have tried changing the parameter in the top-level SIMFACTORY directory under the job directory. However, looking at the parfile in the new checkpoint directory, it still bore the old value. We even went as far as changing the parameter in all parfiles in the previous checkpoint subdirectory, in addition to changing it in the top-level SIMFACTORY directory.
In any case, if you also suggest that then I'm going to give this a try again over the weekend (or on Monday - our computing center is on a shut down for the weekend).
Hi Dumsani,
The top-level simfactory directory I am talking about is:
mysim/ SIMFACTORY/ <------ This one par/ mysim.par output-0000/ SIMFACTORY output-0001/ SIMFACTORY/ ...
i.e. there are SIMFACTORY directories inside each restart, and there is one at the very top level of the simulation. If you change the parameter file in this one, it should be copied into the output-NNNN/SIMFACTORY/par directory when the new restart is created. That was my understanding of how it works, though I admit that I have not used SimFactory 2 for a while. Do other people use this mechanism and can confirm that this is how it works?
-- Ian Hinder http://members.aei.mpg.de/ianhin
On Fri, Sep 30, 2016 at 04:04:10PM +0200, dumsani wrote:
Cactus/Simfactory? I notice that upon recovery (in the next re-start) Simfactory reads the parameters from a checkpoint file, not from any of the parfiles dumped within the job directory.
Cactus does read parameters from the checkpoint, but they can and will be overwritten by parameters given in the parameter file for the restart. The only remaining question would be which file simfactory uses for the restart, and how to best make changes to that, or how to specify a new one.
Frank
users@lists.einsteintoolkit.org
-
dumsani -
Frank Loeffler -
Ian Hinder