[ET Trac] [Einstein Toolkit] #64: Refactor/redesign archiving

Einstein Toolkit trac-noreply at einsteintoolkit.org
Thu Aug 4 09:57:34 CDT 2011


#64: Refactor/redesign archiving
-------------------------+--------------------------------------------------
  Reporter:  mthomas     |       Owner:  mthomas
      Type:  defect      |      Status:  new    
  Priority:  minor       |   Milestone:         
 Component:  SimFactory  |     Version:         
Resolution:              |    Keywords:         
-------------------------+--------------------------------------------------

Comment (by hinder):

 If we make very large tar files before archiving, on lustre filesystems
 this might lead to a single storage target becoming full.  We can set the
 stripe count of the tar file to ensure it is spread across multiple
 storage targets to avoid this problem.

   lfs setstripe -c -1 <filename>

 Since each machine probably has its own archiving system, we will want to
 be able to choose an archiving script for each machine in the mdb.  We
 could have standard ones for, e.g., TSM.  This could detect if it was
 running on lustre and if so, set the stripe count.

 When more than one person in a group works on a project, each of them
 should be given access to restore the archive.  There are TSM commands to
 do this.  There should be a mechanism in simfactory for deciding who to
 give access to, and this should be done by default.  This could be
 overridden on a per-simulation basis.  Something like "--archive-access
 user1,user2,user3" and "archive-access = user1,user2,user3".

 Some ideas from Erik:

 * High level commands provided by simfactory:

     1. archive a simulation
     2. list archived simulations
     3. restore a simulation
     4. delete an archive

 * When compressing files, often gzip --fast is much faster than the
 default options, and the loss of compression is fairly small. If
 compression time becomes a bottleneck I would try this.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/64#comment:3>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit


More information about the Trac mailing list