<div dir="ltr"><div>The Einstein Toolkit is a complex piece of software. Almost by definition, and certainly by design, the Cactus framework chooses to build (and thus depend) upon other software, rather than re-inventing the wheel. Thus we have to deal with linking against external software, such as MPI and HDF5, and a long rat's tail of others (BLAS, FFTW, GSL, hwloc, LAPACK, OpenSSL, PAPI, PETSc, zlib). It is an unfortunate reality of current HPC programming that using libraries remains a very difficult task.</div><div><br></div><div>Our current way of handling these in Cactus is two-fold:</div><div>(1) there is a flexible mechanism to find, check for, or manually specify the location of external libraries</div><div>(2) external libraries can be bundled in a Cactus thorn, and be built automatically</div><div>(3) Cactus can choose automatically between these two</div><div><br></div><div>I claim that the current mechanism does not work well, for at least the following reasons:</div><div>- if Cactus builds an external library, it does so too often (once per configuration or executable, not once per system or compiler version)</div><div>- the current convenience mechanism it overly complex, in the sense that people don't understand any more under what conditions an external library is built; Cactus does things differently from what people expect, and errors go unnoticed</div><div>- bundling source code for external libraries makes for large thorns, even if it is not used</div><div>- the current implementation is complex and difficult to maintain, for a variety of reasons that have been extensively discussed</div><div>- there are several other (minor) problems that are nonetheless difficult to solve</div><div><br></div><div>Furthermore, people have different preferences regarding building or using external libraries, ranging from making it simple for new users, to reducing download time/disk space, reducing build time, to obtaining the best possible performance. These conflicting goals (all valid!) make it difficult to work towards a better solution.</div><div><br></div><div>To solve this problem, I make the following radical suggestion:</div><div><br></div><div>>>> We stop building external libraries as part of building a Cactus executable. <<<</div><div><br></div><div>It was a good and promising idea, but it failed to deliver; it is causing more problems than it solves.</div><div><br></div><div><br></div><div><br></div><div>In the new system, there will be two kinds of external libraries:</div><div>- like MPI or HDF5: an external library must be present that Cactus uses</div><div>- like LORENE: we always build our own (more on this below)</div><div><br></div><div>Clearly, we still must make it easy to install a missing external library. Building PETSc is not easy, and then pointing Cactus to a self-built PETSc is not easy either. There are various options to build an external library outside Cactus:</div><div>(1) apt-get install or similar (for local machines)</div><div>(2) module/softenv (on many HPC systems)</div><div>(3) the ET maintainers build it, and make it publicly available, on a set of "important" machines</div><div>(4) manual build (configure; make; make install) -- often not difficult, if brief (ten lines) of instructions are given<br></div><div>(5) we provide thorns and parameter files that automate this, as in "exe/cactus_sim build-hdf5.par"; these would need to be run explicitly once per user on every system</div><div>(6) using a professional package manager that doesn't require root access, such as Spack <<a href="https://github.com/LLNL/spack">https://github.com/LLNL/spack</a>></div><div><br></div><div><br></div><div><br></div><div>Regarding thorns like LORENE: We should switch from distributing the LORENE source code as tarball to keeping the unpacked source code in the thorn, with a set of patches applied. In this way, LORENE can be built as regular Cactus thorn, together with other Cactus source code, and will be linked in a natural way -- no configuration magic necessary at all. All we need is a makefile that lists the LORENE source files.</div><div><br></div><div>-erik</div><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@gmail.com" target="_blank">schnetter@gmail.com</a>> <a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br></div>
</div>