<div dir="ltr">Hello Ian, <div><br></div><div>yes, this has happened to me.. I didn't use a shred checkpoint directory, and some runs have started from the initial data again..deleting all folders output-NNNN without a valid termination checkpoint will remedy this?</div>
<div><br></div><div>to use a shared checkpoint directory for future runs: is it sufficient to just use </div><div><br></div><div>io::checkpoint_dir = <span style="font-family:arial,sans-serif;font-size:13px">"../checkpoints"</span><br>
</div><div><font face="arial, sans-serif">io::recover_dir = </font><span style="font-family:arial,sans-serif;font-size:13px">"../checkpoints"</span><br></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">??</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">best wishes,</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Vassili</span></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Jul 28, 2014 at 12:33 PM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
On 23 Jul 2014, at 18:32, Vassilios Mewes <<a href="mailto:vassilios.mewes@uv.es">vassilios.mewes@uv.es</a>> wrote:<br>
<br>
> Hello all,<br>
><br>
> a simulation has crashed without checkpointing (there was a filesystem error on the cluster)<br>
><br>
> how can i restart it? do I need to delete the uncompleted output-xxxx and output-xxx-active folder? or is that not necessary and simfactory will automatically find the latest valid checkpoint in simulation time and restart from there?<br>
<br>
</div></div>SimFactory is supposed to recover from this situation gracefully. However, I have in the past seen it notice that there are no checkpoint files in the last restart, and then start again from the initial data. Perhaps somebody forgot to write a test case for this situation. This is not a problem if you use a checkpoint directory shared between restarts (i.e. "../checkpoints"). If you are not using a shared checkpoint directory, I recommend deleting the output-NNNN and output-NNNN-active directories/links for the failed restarts.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Ian Hinder<br>
<a href="http://numrel.aei.mpg.de/people/hinder" target="_blank">http://numrel.aei.mpg.de/people/hinder</a><br>
<br>
<br>
</font></span></blockquote></div><br></div>