[ET Trac] [Einstein Toolkit] #899: Push testsuite results to a git repository

[Einstein Toolkit]

#899: Push testsuite results to a git repository
--------------------------+-------------------------------------------------
  Reporter:  eschnett     |       Owner:     
      Type:  enhancement  |      Status:  new
  Priority:  major        |   Milestone:     
 Component:  Cactus       |     Version:     
Resolution:               |    Keywords:     
--------------------------+-------------------------------------------------

Comment (by hinder):

 >Would it make sense to have each test run as a single simulation with
 multiple restarts? This would fit better into the Simfactory scheme, but
 would prevent running them in parallel.

 At the moment, compilation on Datura takes about 10 minutes (on a good
 day), and each test run also takes about 10 minutes.  So yes, it would be
 useful to run them in parallel (30 mins down to 20 mins), but not
 essential.  Performance on other machines will vary, but I expect the
 compilation time to dominate (we have a relatively fast home filesystem).
 This is, of course, an abuse of the simfactory restart mechanism.  We
 could alternatively teach simfactory to run multiple test sets in a single
 simulation.  It already has special code for dealing with test suites.
 This would need some tweaking of mpirun/nprocs settings.

 >I usually have branches for configurations (not machines). That means
 that e.g. damiana-gcc and damiana-intel would be in different branches.

 >It is possible to expire old runs. This would probably make it necessary
 to not link the different commits into a sequence (but rather have a
 multi-headed repository). Each commit would also be tagged. Removing the
 tag would then expire a run. This may not be a good idea, because it may
 confuse many tools. I don't do this in Formaline.

 Having separate heads would indeed enable run expiry.  Why would tools get
 confused?  What we could also do is start a new head every time the tests
 pass successfully.  We are not usually interested in history before that
 point any more.  We could then expire old heads periodically if we wanted
 to.

 >I just checked, and a single test result uses 8 MB. This indicates an
 incremental cost of about 2 MB per test run. I assume you run "git gc"
 from time to time, and that 363 MB is measured after a gc?

 Yes, I ran a repack and a gc before measuring the repo.  Does your 8 MB
 include both 1 proc and 2 proc results?  That is quite a large incremental
 cost.  It should be much smaller than that.  It is probably not an
 incremental cost, but the cost of regenerating test data from time to
 time.  There are also one or two very fishy tests which generate different
 numerical results each time they are run (but within tolerance).  Having
 test results as separate commits in the same branch allowed me to detect
 this sort of behaviour.

 >I would store the output data as well. If we run out of space, we can
 reduce the size (but moving some of the files to a new repository).

 Or just starting a new repository in a state where all the tests pass,
 which could correspond to a 6-monthly release.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/899#comment:3>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit