The Einstein Toolkit is a complex piece of software. Almost by definition, and certainly by design, the Cactus framework chooses to build (and thus depend) upon other software, rather than re-inventing the wheel. Thus we have to deal with linking against external software, such as MPI and HDF5, and a long rat's tail of others (BLAS, FFTW, GSL, hwloc, LAPACK, OpenSSL, PAPI, PETSc, zlib). It is an unfortunate reality of current HPC programming that using libraries remains a very difficult task.
Our current way of handling these in Cactus is two-fold: (1) there is a flexible mechanism to find, check for, or manually specify the location of external libraries (2) external libraries can be bundled in a Cactus thorn, and be built automatically (3) Cactus can choose automatically between these two
I claim that the current mechanism does not work well, for at least the following reasons: - if Cactus builds an external library, it does so too often (once per configuration or executable, not once per system or compiler version) - the current convenience mechanism it overly complex, in the sense that people don't understand any more under what conditions an external library is built; Cactus does things differently from what people expect, and errors go unnoticed - bundling source code for external libraries makes for large thorns, even if it is not used - the current implementation is complex and difficult to maintain, for a variety of reasons that have been extensively discussed - there are several other (minor) problems that are nonetheless difficult to solve
Furthermore, people have different preferences regarding building or using external libraries, ranging from making it simple for new users, to reducing download time/disk space, reducing build time, to obtaining the best possible performance. These conflicting goals (all valid!) make it difficult to work towards a better solution.
To solve this problem, I make the following radical suggestion:
We stop building external libraries as part of building a Cactus
executable. <<<
It was a good and promising idea, but it failed to deliver; it is causing more problems than it solves.
In the new system, there will be two kinds of external libraries: - like MPI or HDF5: an external library must be present that Cactus uses - like LORENE: we always build our own (more on this below)
Clearly, we still must make it easy to install a missing external library. Building PETSc is not easy, and then pointing Cactus to a self-built PETSc is not easy either. There are various options to build an external library outside Cactus: (1) apt-get install or similar (for local machines) (2) module/softenv (on many HPC systems) (3) the ET maintainers build it, and make it publicly available, on a set of "important" machines (4) manual build (configure; make; make install) -- often not difficult, if brief (ten lines) of instructions are given (5) we provide thorns and parameter files that automate this, as in "exe/cactus_sim build-hdf5.par"; these would need to be run explicitly once per user on every system (6) using a professional package manager that doesn't require root access, such as Spack https://github.com/LLNL/spack
Regarding thorns like LORENE: We should switch from distributing the LORENE source code as tarball to keeping the unpacked source code in the thorn, with a set of patches applied. In this way, LORENE can be built as regular Cactus thorn, together with other Cactus source code, and will be linked in a natural way -- no configuration magic necessary at all. All we need is a makefile that lists the LORENE source files.
-erik