#180: Reduce disk space used by checkpoints --------------------------+------------------------------------------------- Reporter: hinder | Owner: mthomas Type: enhancement | Status: new Priority: major | Milestone: Component: SimFactory | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by eschnett):
SimFactory's philosophy is that each restart is independent of the other restarts, like a snapshop of the simulation in time. The hard-linking mechanism was introduced to prevent Cactus from (accidentally?) deleting data from old restarts.
One problem in long-running simulations is that (a) one may accidentally delete too many checkpoints, so that the simulation becomes unusable. Another problem is that, if an error in the simulation is detected and if all old checkpoints have been deleted, it is impossible to "go back in time" (using checkpoints left every so many time steps).
My current idea is to archive old restarts, and then delete them (or parts of them) locally. In this way, no data have to be deleted. This depends on the availability of archival storage, but this should be present -- we mostly have not been using it because this is somewhat awkward: it takes a long time, different systems have different archives, and it is cumbersome to find out which files have been archived. However, it should in principle be straightforward to automatically copy older restarts into an archive.