#316: Checkpoint recovery nonfunctional ------------------------+--------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Keywords: regression | ------------------------+--------------------------------------------------- Checkpoint recovery is nonfunctional in SimFactory 2 (it has broken since it was last fixed in ticket #60).
Using the attached parameter file, I submit a simulation on Datura:
simfactory2/bin/sim --machine datura --config sim2_datura create-submit parfiles/cptest.par 12 1:00:00
This parameter file terminates the Cactus run after 1 minute and dumps a checkpoint file. I then manually remove the output-0000-active symlink, as the automatic cleanup in the main() function is cleaning up restarts that are attempting to run, so I have disabled it, and manual cleanup doesn't work (see ticket #315).
I then resubmit the simulation
simfactory2/bin/sim --machine datura submit parfiles/cptest.par
and observe that the checkpoint files from the first restart are never hardlinked into the output directory. The job does not recover, and instead starts from initial data.
Log file is attached.
Looking at the code, it appears that the checkpoint linking is conditional on the from-restart-id parameter being passed to simfactory, which I think is something to do with job-chaining. I can't see anywhere in the code which sets this option, so this is probably why the linking is not happening.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by hinder):
Workaround: recovery can be made to work if you add the options
--from-restart-id=<n> --recover
to the submit command, where <n> is the number of the restart that you want to recover from.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by mthomas):
Ian, I have committed a fix for this. Please test without using --from- restart-id or --recover.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by hinder):
I cannot get this to work on my laptop. It still doesn't seem to be linking the checkpoint files into the new restart directory. Do you think it depends on the machine that I run on? I have tested this on my laptop. What machine have you tested this on, so that I can try on the same machine?
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by mthomas):
I have attached a patch to fix checkpointing. please test.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by eschnett):
The patch looks good; since Ian confirmed that it solves the problem, please apply it.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by hinder):
It looks like it only attempts to recover from the previous restart. What happens if that restart never ran, perhaps because it was deleted from the queue, and hence there are no checkpoint files in it? I think simfactory should try to recover from the last restart which actually has checkpoint files, not just the last restart. If there are no restarts with checkpoint files, then it should not try to recover.
I don't think the patch should be applied until it works for the case where a job never ran, and hence there are no checkpoint files in the last few restarts.
Aside: all of this would have been trivial if we used a single common directory for checkpoint files, common across restarts. All the logic is present in Cactus already.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by eschnett):
Let's discuss this on the mailing list. Using a common checkpoint directory may be a good idea.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by mthomas):
I have attached a new patch that finds all checkpointing files and links them. Please review.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by eschnett):
This patch has several problems:
We just agreed on the mailing list to store checkpoint files in a directory "checkpoints" that is at the same level as the "output-NNNN" directories. These checkpoint files therefore do not fit the pattern you are expecting, and lead to warnings. Instead, these checkpoint files should be ignored. Instead of looking for checkpoint files in the whole simulation directory, it may be better to look in the individual restart directories.
The code to replace the "output-NNNN" patterns is too complex. Instead of pattern matching and then manual string operations, a direct regexp replacement would be simpler and safer.
The pattern matching code does not check where in the path name the pattern "output-NNNN" exists. If a simulation is called "output", this leads to problems.
The message "linking file XXX" is printed even if the linking step does not actually happen.
The code to check whether a checkpoint file is found twice only looks at file names. This is fragile and can hide problems, for example when two restarts use different numbers of processes. Instead, the incoming list of checkpoint files should be pruned. This would also allow an explicit choice of whether newer or older checkpoint files should be chosen if they exist in several restarts: the current code presumably chooses the older ones, whereas we may want to use the newer ones.
The fallback call to "shutil.copyfile" overwrites existing file contents. This is a serious problem if the existing file is a hard link (because this will then overwrite the file content). The destination needs to be unlinked first.
This code links all checkpoint files. Only checkpoint files from the last iteration should be linked.
Overall, the code says "checkpoint" when it should say "recover" in several places.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: ET_2011_05 Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+-------------------------------------------------- Changes (by knarf):
* milestone: => ET_2011_05
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: ET_2011_11 Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+-------------------------------------------------- Changes (by knarf):
* milestone: ET_2011_05 => ET_2011_11
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: new Priority: blocker | Milestone: ET_2011_11 Component: SimFactory | Version: Resolution: | Keywords: regression -------------------------+--------------------------------------------------
Comment (by eschnett):
Note: I have just successfully submitted three restarts of a simulation on Damiana, using a global "checkpoints" directory.
#316: Checkpoint recovery nonfunctional -------------------------+-------------------------------------------------- Reporter: hinder | Owner: mthomas Type: defect | Status: closed Priority: blocker | Milestone: ET_2011_11 Component: SimFactory | Version: Resolution: fixed | Keywords: regression -------------------------+-------------------------------------------------- Changes (by eschnett):
* status: new => closed * resolution: => fixed
Comment:
Note: In my comment above, I meant "finished" instead of "submitted".
I have also just successfully finished three restarts of a simulation on Damiana, without using a global "checkpoints" directory.
Since the problem seems to not exist any more, I am closing this ticket.
trac@lists.einsteintoolkit.org
-
Einstein Toolkit