[ET Trac] [Einstein Toolkit] #488: Reduce space taken by Formaline tarballs

[Einstein Toolkit]

#488: Reduce space taken by Formaline tarballs
--------------------------+-------------------------------------------------
  Reporter:  hinder       |       Owner:           
      Type:  enhancement  |      Status:  review   
  Priority:  minor        |   Milestone:           
 Component:  Cactus       |     Version:           
Resolution:               |    Keywords:  Formaline
--------------------------+-------------------------------------------------

Comment (by eschnett):

 We are discussing Simfactory here. The Formaline tarballs are only one of
 the issues. With this patch, Simfactory will combine all possible large
 files via hardlinks, not just Formaline tarballs. This will be done only
 within each simulation.

 One use case of Simfactory is to run benchmarks. This involves producing
 many small simulations, and the overhead of copying executables is
 significant and prohibitive, as the benchmark output is much smaller than
 the size of the executables.

 I see and understand the problem with hardlinks. One crazy solution could
 involve git, which internally automatically collapses identical files. Of
 course, having both the repository and a checkout present will double the
 disk space (for compressed files) -- is there a way to use git without
 local repository? We could also just wait until deduplicating file systems
 are more common in HPC.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/488#comment:7>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit