[ET Trac] [Einstein Toolkit] #1772: Simfactory: potentially serious problem with CACHE directory in the simulations directory

[Einstein Toolkit]

#1772: Simfactory: potentially serious problem with CACHE directory in the
simulations directory
-------------------------+--------------------------------------------------
  Reporter:  bmundim     |       Owner:                     
      Type:  defect      |      Status:  new                
  Priority:  critical    |   Milestone:  ET_2015_05         
 Component:  SimFactory  |     Version:  development version
Resolution:              |    Keywords:  CACHE              
-------------------------+--------------------------------------------------

Comment (by eschnett):

 Yes, the CACHE directory is necessary if you are running many similar
 simulations. This can happen e.g. during benchmarking. While most HPC
 systems can handle a large number of executables, there are some that
 cannot, and where one runs out of quota.

 The cache works slightly differently than you describe. First, it uses
 hard links, not soft (symbolic) links. The cache is just this, a cache --
 the actual executables are safely stored in the simulation directories,
 and are never modified. Here is what actually happens when a simulation is
 created:

 1. Check cache whether it has the right executable. If not, ignore cache.
 2. If cache is good, create a hard link from cache to simulation
 directory.
 3. Check simulation directory if it now has the right executable, since
 the cache may have changed in the mean time. If no, delete hard link
 again.
 4. If simulation directory does not have an executable, copy it from the
 Cactus build directory.
 5. The simulation directory now has the correct executable.
 6. If we are not using a hard link from the cache, then delete the cache
 file, and create a new hard link from the simulation directory to the
 cache.

 This guarantees that simulations always use the correct executable, and
 that a simulation's executable is never overwritten. Also, there are
 fallbacks in place in case any of the operations fails (e.g. creating a
 hard link).

 What is actually a problem currently is that some external libraries that
 are built are dynamic libraries, which can change or go away when one
 rebuilds. These are not copied into the simulation directory. We try to
 enforce using static libraries for this reason, but I'm not sure whether
 this is the case for all external libraries.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1772#comment:1>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit