#488: Reduce space taken by Formaline tarballs --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: review Priority: minor | Milestone: Component: Cactus | Version: Resolution: | Keywords: Formaline --------------------------+-------------------------------------------------
Comment (by eschnett):
We are discussing Simfactory here. The Formaline tarballs are only one of the issues. With this patch, Simfactory will combine all possible large files via hardlinks, not just Formaline tarballs. This will be done only within each simulation.
One use case of Simfactory is to run benchmarks. This involves producing many small simulations, and the overhead of copying executables is significant and prohibitive, as the benchmark output is much smaller than the size of the executables.
I see and understand the problem with hardlinks. One crazy solution could involve git, which internally automatically collapses identical files. Of course, having both the repository and a checkout present will double the disk space (for compressed files) -- is there a way to use git without local repository? We could also just wait until deduplicating file systems are more common in HPC.