#64: Refactor/redesign archiving -------------------------+-------------------------------------------------- Reporter: mthomas | Owner: mthomas Type: defect | Status: new Priority: minor | Milestone: Component: SimFactory | Version: Resolution: | Keywords: -------------------------+--------------------------------------------------
Comment (by hinder):
If we make very large tar files before archiving, on lustre filesystems this might lead to a single storage target becoming full. We can set the stripe count of the tar file to ensure it is spread across multiple storage targets to avoid this problem.
lfs setstripe -c -1 <filename>
Since each machine probably has its own archiving system, we will want to be able to choose an archiving script for each machine in the mdb. We could have standard ones for, e.g., TSM. This could detect if it was running on lustre and if so, set the stripe count.
When more than one person in a group works on a project, each of them should be given access to restore the archive. There are TSM commands to do this. There should be a mechanism in simfactory for deciding who to give access to, and this should be done by default. This could be overridden on a per-simulation basis. Something like "--archive-access user1,user2,user3" and "archive-access = user1,user2,user3".
Some ideas from Erik:
* High level commands provided by simfactory:
1. archive a simulation 2. list archived simulations 3. restore a simulation 4. delete an archive
* When compressing files, often gzip --fast is much faster than the default options, and the loss of compression is fairly small. If compression time becomes a bottleneck I would try this.