#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: hinder Type: defect | Status: accepted Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+---------------------------------------
Comment (by hinder):
I have seen something similar to this recently as well. The test GRHydro_test_tov_ppm_ML actually passed on one run and failed when it was run again. The only commit between these two modified unrelated test data and parameter files, and couldn't possibly have changed the behaviour of GRHydro.
The first run was http://git.barrywardell.net/EinsteinToolkitTestResults.git/tree/10647caa473a... and the test passed (http://git.barrywardell.net/EinsteinToolkitTestResults.git/blob/10647caa473a...). The second run was http://git.barrywardell.net/EinsteinToolkitTestResults.git/tree/d7442650e1d7... and the tests failed (http://git.barrywardell.net/EinsteinToolkitTestResults.git/blob/d7442650e1d7...).
The only commit between these two runs was https://github.com/barrywardell/EinsteinExact/commit/26f0b04b7af196673b10043... which could not change the results for the GRHydro test.
Note that this test has been consistently passing when run using the old test system (http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/).
The executables are available in datura:/lustre/datura/ianhin/projects/temp.