[ET Trac] [Einstein Toolkit] #522: Improve consistency and performance of sync/build/create

[Einstein Toolkit]

#522: Improve consistency and performance of sync/build/create
-------------------------+--------------------------------------------------
 Reporter:  hinder       |       Owner:  eschnett
     Type:  enhancement  |      Status:  new     
 Priority:  minor        |   Milestone:          
Component:  SimFactory   |     Version:          
 Keywords:               |  
-------------------------+--------------------------------------------------
 One of the design goals of SimFactory is to make it very difficult for a
 user to make mistakes or end up with inconsistent results.  Currently, if
 a user modifies a source file and submits a simulation without remembering
 to rebuild the configuration, the old version of the configuration will be
 used.  SimFactory could scan all the source files which would be used for
 a build and warn the user if there are new source files.  Optionally it
 could also rebuild the configuration automatically.  This check might be
 better implemented in Cactus, which in principle has the dependency
 information already.  It is functionally equivalent to performing a build
 before every create, so maybe the best approach would be to make the
 Cactus build process much faster in the case where no files have been
 modified.  This may not be possible.

 A related idea is to ensure that remote configurations are synced before
 being built, and parameter files are synced before simulations are
 created.   One could record the time of the last successful sync
 (separately of the source tree and the parameter files) to each machine,
 and look for files which have been modified since that time.  If there are
 such files, a sync could be initiated, or the user warned.  Statting files
 on a laptop should be very fast.  Statting files on Kraken can take
 minutes or hours.  This information could also be used to restrict the
 list of files synced, to avoid having to stat every file in the source
 tree on the remote system.  This assumes that the only modifications of
 the source tree on the remote system are due to simfactory on the local
 system.  For me, this is the case 99.999% of the time.

 I attach a script that I wrote to interact conveniently with remote
 systems.  It performs a sync of either the source tree or the parameter
 files depending on the simfactory command you want to run.  It allows you
 to use the syntax

   <machine> <command> <args>

 instead of

   sim --remote <machine> <command> <args>

 e.g.

   damiana list-simulations

 The script should be somewhere on your path, and symlinks named after
 machines should be created pointing to it.  For example,

   cp machine .../bin
   chmod u+x .../bin/sim/machine
   cd .../bin
   ln -s machine datura
   ln -s machine damiana
   ln -s machine kraken

   cd ~/Cactus

   datura sync
   datura create-submit ...
   datura build

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/522>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit