#1568: Reduce overhead of Formaline --------------------------+------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: major | Milestone: Component: Cactus | Version: development version Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by eschnett):
That may be a nice idea. However, I would be careful especially with git repositories where commits or branches may exist only locally.
I don't think that storing the source code should be as expensive as it is. If we exclude the ExternalLibraries tarballs, then each source file is already read by the compiler, and converted to an object file. The executable (without tarballs) is larger than the tarballs. I don't understand why generating them is so expensive. My current assumption is that this process is only insufficiently parallelized by make, and that e.g. creating the tarballs requires a lot of disk I/O with little CPU action and happens all at once. Spreading this out over a longer time, and having tar read the source files near the time when the compiler already reads them, should improve performance considerably.
We may also further improve performance by not storing the temporary tarballs on disk before using perl to convert them to a C source file; this could happen in one step, and the result could be fed directly to the compiler. This would require writing a small C program to do so, based on zlib for compression.
Regarding the git repositories: There is currently one git repo per executable, which is a serialization choking point. Using one git repo per thorn would be much faster (since parallel), and combining these (via branches? merging? simple sequential commits?) may be faster than a single, large commit.