#1772: Simfactory: potentially serious problem with CACHE directory in the simulations directory -------------------------+-------------------------------------------------- Reporter: bmundim | Owner: Type: defect | Status: new Priority: critical | Milestone: ET_2015_05 Component: SimFactory | Version: development version Resolution: | Keywords: CACHE -------------------------+--------------------------------------------------
Comment (by eschnett):
Yes, the CACHE directory is necessary if you are running many similar simulations. This can happen e.g. during benchmarking. While most HPC systems can handle a large number of executables, there are some that cannot, and where one runs out of quota.
The cache works slightly differently than you describe. First, it uses hard links, not soft (symbolic) links. The cache is just this, a cache -- the actual executables are safely stored in the simulation directories, and are never modified. Here is what actually happens when a simulation is created:
1. Check cache whether it has the right executable. If not, ignore cache. 2. If cache is good, create a hard link from cache to simulation directory. 3. Check simulation directory if it now has the right executable, since the cache may have changed in the mean time. If no, delete hard link again. 4. If simulation directory does not have an executable, copy it from the Cactus build directory. 5. The simulation directory now has the correct executable. 6. If we are not using a hard link from the cache, then delete the cache file, and create a new hard link from the simulation directory to the cache.
This guarantees that simulations always use the correct executable, and that a simulation's executable is never overwritten. Also, there are fallbacks in place in case any of the operations fails (e.g. creating a hard link).
What is actually a problem currently is that some external libraries that are built are dynamic libraries, which can change or go away when one rebuilds. These are not copied into the simulation directory. We try to enforce using static libraries for this reason, but I'm not sure whether this is the case for all external libraries.