[ET Trac] [Einstein Toolkit] #64: Refactor/redesign archiving
Einstein Toolkit
trac-noreply at einsteintoolkit.org
Thu Aug 4 09:57:34 CDT 2011
#64: Refactor/redesign archiving
-------------------------+--------------------------------------------------
Reporter: mthomas | Owner: mthomas
Type: defect | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
-------------------------+--------------------------------------------------
Comment (by hinder):
If we make very large tar files before archiving, on lustre filesystems
this might lead to a single storage target becoming full. We can set the
stripe count of the tar file to ensure it is spread across multiple
storage targets to avoid this problem.
lfs setstripe -c -1 <filename>
Since each machine probably has its own archiving system, we will want to
be able to choose an archiving script for each machine in the mdb. We
could have standard ones for, e.g., TSM. This could detect if it was
running on lustre and if so, set the stripe count.
When more than one person in a group works on a project, each of them
should be given access to restore the archive. There are TSM commands to
do this. There should be a mechanism in simfactory for deciding who to
give access to, and this should be done by default. This could be
overridden on a per-simulation basis. Something like "--archive-access
user1,user2,user3" and "archive-access = user1,user2,user3".
Some ideas from Erik:
* High level commands provided by simfactory:
1. archive a simulation
2. list archived simulations
3. restore a simulation
4. delete an archive
* When compressing files, often gzip --fast is much faster than the
default options, and the loss of compression is fairly small. If
compression time becomes a bottleneck I would try this.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/64#comment:3>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
More information about the Trac
mailing list