[ET Trac] [Einstein Toolkit] #1568: Reduce overhead of Formaline

[Einstein Toolkit]

#1568: Reduce overhead of Formaline
--------------------------+-------------------------------------------------
  Reporter:  hinder       |       Owner:                     
      Type:  enhancement  |      Status:  new                
  Priority:  major        |   Milestone:                     
 Component:  Cactus       |     Version:  development version
Resolution:               |    Keywords:                     
--------------------------+-------------------------------------------------

Comment (by eschnett):

 That may be a nice idea. However, I would be careful especially with git
 repositories where commits or branches may exist only locally.

 I don't think that storing the source code should be as expensive as it
 is. If we exclude the ExternalLibraries tarballs, then each source file is
 already read by the compiler, and converted to an object file. The
 executable (without tarballs) is larger than the tarballs. I don't
 understand why generating them is so expensive. My current assumption is
 that this process is only insufficiently parallelized by make, and that
 e.g. creating the tarballs requires a lot of disk I/O with little CPU
 action and happens all at once. Spreading this out over a longer time, and
 having tar read the source files near the time when the compiler already
 reads them, should improve performance considerably.

 We may also further improve performance by not storing the temporary
 tarballs on disk before using perl to convert them to a C source file;
 this could happen in one step, and the result could be fed directly to the
 compiler. This would require writing a small C program to do so, based on
 zlib for compression.

 Regarding the git repositories: There is currently one git repo per
 executable, which is a serialization choking point. Using one git repo per
 thorn would be much faster (since parallel), and combining these (via
 branches? merging? simple sequential commits?) may be faster than a
 single, large commit.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1568#comment:3>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit