[Users] Thoughts on external libraries

[Erik Schnetter]

The Einstein Toolkit is a complex piece of software. Almost by definition,
and certainly by design, the Cactus framework chooses to build (and thus
depend) upon other software, rather than re-inventing the wheel. Thus we
have to deal with linking against external software, such as MPI and HDF5,
and a long rat's tail of others (BLAS, FFTW, GSL, hwloc, LAPACK, OpenSSL,
PAPI, PETSc, zlib). It is an unfortunate reality of current HPC programming
that using libraries remains a very difficult task.

Our current way of handling these in Cactus is two-fold:
(1) there is a flexible mechanism to find, check for, or manually specify
the location of external libraries
(2) external libraries can be bundled in a Cactus thorn, and be built
automatically
(3) Cactus can choose automatically between these two

I claim that the current mechanism does not work well, for at least the
following reasons:
- if Cactus builds an external library, it does so too often (once per
configuration or executable, not once per system or compiler version)
- the current convenience mechanism it overly complex, in the sense that
people don't understand any more under what conditions an external library
is built; Cactus does things differently from what people expect, and
errors go unnoticed
- bundling source code for external libraries makes for large thorns, even
if it is not used
- the current implementation is complex and difficult to maintain, for a
variety of reasons that have been extensively discussed
- there are several other (minor) problems that are nonetheless difficult
to solve

Furthermore, people have different preferences regarding building or using
external libraries, ranging from making it simple for new users, to
reducing download time/disk space, reducing build time, to obtaining the
best possible performance. These conflicting goals (all valid!) make it
difficult to work towards a better solution.

To solve this problem, I make the following radical suggestion:

>>> We stop building external libraries as part of building a Cactus
executable. <<<

It was a good and promising idea, but it failed to deliver; it is causing
more problems than it solves.



In the new system, there will be two kinds of external libraries:
- like MPI or HDF5: an external library must be present that Cactus uses
- like LORENE: we always build our own (more on this below)

Clearly, we still must make it easy to install a missing external library.
Building PETSc is not easy, and then pointing Cactus to a self-built PETSc
is not easy either. There are various options to build an external library
outside Cactus:
(1) apt-get install or similar (for local machines)
(2) module/softenv (on many HPC systems)
(3) the ET maintainers build it, and make it publicly available, on a set
of "important" machines
(4) manual build (configure; make; make install) -- often not difficult, if
brief (ten lines) of instructions are given
(5) we provide thorns and parameter files that automate this, as in
"exe/cactus_sim build-hdf5.par"; these would need to be run explicitly once
per user on every system
(6) using a professional package manager that doesn't require root access,
such as Spack <https://github.com/LLNL/spack>



Regarding thorns like LORENE: We should switch from distributing the LORENE
source code as tarball to keeping the unpacked source code in the thorn,
with a set of patches applied. In this way, LORENE can be built as regular
Cactus thorn, together with other Cactus source code, and will be linked in
a natural way -- no configuration magic necessary at all. All we need is a
makefile that lists the LORENE source files.

-erik

-- 
Erik Schnetter <schnetter at gmail.com>
http://www.perimeterinstitute.ca/personal/eschnetter/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20160608/e1ba32bd/attachment.html