#1283: Missing data in HDF5 files ---------------------+------------------------------------------------------ Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Cactus | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by eschnett):
I would not use "tmp" to indicate files are currently incomplete. What about "incomplete"? Or "$$$"?
One either needs to store the data belonging to a simulation, or re-run the simulation if there is a problem. If archives can't handle checkpoint files, and if local data disks aren't large enough to keep checkpoint files, then re-running is the correct response. People need to be aware of this, and in this mode, all but the last (verified to be safely usable) checkpoint files should be deleted, probably semi-automatically.
If a data disk has 300 Tbyte, then I would expect an archive to have 3,000 Tbyte. Alternatively, one could design a system that has 100 Tbyte data disk and 1,000 Tbyte usable archive space (10 Gbit interconnect?). Currently, since no one uses archive except to stash data they don't intend to look at any more, the archive is difficult to use, and there is a large pressure to increase data disks, further reducing the need to use an archive, and thus reducing pressure to make it easier to use.
I meant checkpoint files. But, thinking about this, it could equally apply to restarts, or at least large output files in restarts.