[ET Trac] [Einstein Toolkit] #899: Push testsuite results to a git repository

[Einstein Toolkit]

#899: Push testsuite results to a git repository
--------------------------+-------------------------------------------------
  Reporter:  eschnett     |       Owner:     
      Type:  enhancement  |      Status:  new
  Priority:  major        |   Milestone:     
 Component:  Cactus       |     Version:     
Resolution:               |    Keywords:     
--------------------------+-------------------------------------------------

Comment (by hinder):

 I have been doing this in the development autotest system.  The system
 runs the tests on both 1 and 2 processes in separate simfactory
 simulations (in the queuing system), waits for them to finish, and then
 commits the output directories (i.e. TEST/<config>) to a results
 repository.  The results repository is hosted on git.barrywardell.net, and
 there is a gitweb interface to it.  You can see this at
 http://git.barrywardell.net/EinsteinToolkitTestResults.git.  Click on
 "tree" to see the results of each run.  This contains the build log, the
 log from the autotest system, and the TEST directory from the 1 and 2
 process runs (including the summary log and all output). In future, I
 would do this on multiple machines, and the results from each machine
 might be pushed to separate branches in the same repository.

 Comparing your suggestions with my approach, there are a couple of
 differences.

 1. I identify a set of simulations run on different numbers of processes
 as a "test run", as this is the logical unit that is always tested
 together.  Having to manage sets of simulations is harder than managing a
 single "test run" composed of multiple simulations.

 2. I find it useful to have a single repository where each commit
 represents a different test run.  This allows you to track any changes
 between runs using standard git tools. If we also had separate branches
 for different machines, then the usual git tools would also allow you to
 compare between branches.  This also means that you cannot expire old runs
 from the repository, and the repository grows in size without limit.  Git
 only stores one copy of identical files, and also does delta compression
 between revisions.  Both of these should significantly mitigate the size
 requirement.  For reference, with 214 test runs, the repository is 363 MB.

 3. I'm not sure if you were intending to store just the summary.log files
 or also the test output data in the repository.  Maybe it would be enough
 to store the log files and the diffs, rather than the output data itself,
 though having the output data has been useful to me in the past

 If this was implemented in Cactus, I would not have to wait for the
 simulations to finish, as once they were submitted, they would be
 autonomous.  This would be a good thing.  In order to have a single commit
 for both 1 and 2 process tests, each simulation could check whether it was
 the last one to run, and only the last one would do the commit.

 git might not be available on the compute nodes where the test suites are
 run.  This is true on Datura.  In principle we could build git as part of
 the ET and store the executable in the simulation directory for systems
 where this was the case.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/899#comment:1>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit