[ET Trac] [Einstein Toolkit] #727: Test output changes every time some tests are run

[Einstein Toolkit]

#727: Test output changes every time some tests are run
------------------------------------+---------------------------------------
  Reporter:  hinder                 |       Owner:  hinder                                       
      Type:  defect                 |      Status:  accepted                                     
  Priority:  major                  |   Milestone:                                               
 Component:  EinsteinToolkit thorn  |     Version:                                               
Resolution:                         |    Keywords:  testsuites QuasiLocalMeasures IDAxiOddBrillBH
------------------------------------+---------------------------------------

Comment (by hinder):

 I have seen something similar to this recently as well.  The test
 GRHydro_test_tov_ppm_ML actually passed on one run and failed when it was
 run again.  The only commit between these two modified unrelated test data
 and parameter files, and couldn't possibly have changed the behaviour of
 GRHydro.

 The first run was
 http://git.barrywardell.net/EinsteinToolkitTestResults.git/tree/10647caa473ab21d17afc6030c2f65189bc5169f
 and the test passed
 (http://git.barrywardell.net/EinsteinToolkitTestResults.git/blob/10647caa473ab21d17afc6030c2f65189bc5169f:/test_2/GRHydro/GRHydro_test_tov_ppm_ML_disable_internal_excision.diffs).
 The second run was
 http://git.barrywardell.net/EinsteinToolkitTestResults.git/tree/d7442650e1d7cfe5dca3a23a0eaa2df734c0035a
 and the tests failed
 (http://git.barrywardell.net/EinsteinToolkitTestResults.git/blob/d7442650e1d7cfe5dca3a23a0eaa2df734c0035a:/test_2/GRHydro/GRHydro_test_tov_ppm_ML.diffs).

 The only commit between these two runs was
 https://github.com/barrywardell/EinsteinExact/commit/26f0b04b7af196673b10043ca6cd44edda7cffaf
 which could not change the results for the GRHydro test.

 Note that this test has been consistently passing when run using the old
 test system
 (http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/).

 The executables are available in
 datura:/lustre/datura/ianhin/projects/temp.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/727#comment:5>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit