#727: Test output changes every time some tests are run -----------------------------------------------------------+---------------- Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH | -----------------------------------------------------------+---------------- Each time a new run of the test suites is performed, the test output of IDAxiOddBrillBH and QuasiLocalMeasures changes a little. These changes are below the tolerances set in the test.ccl files, so the tests pass, but there should be no change at all when the tests are run on the same machine.
An example of the difference from one test run to the next is shown at http://git.barrywardell.net/EinsteinToolkitTestResults.git/blobdiff/95fe088e....
Another example, from QuasiLocalMeasures, is at http://git.barrywardell.net/EinsteinToolkitTestResults.git/blobdiff/95fe088e... /qlm-ks-shifted/admbase::metric.average.asc.
The only thorns to exhibit this behaviour are IDAxiOddBrillBH and QuasiLocalMeasures. All other test data remains unchanged from one test run to the next. Note that the two test runs will be performed with different executables, since a new configuration is created for each run. I think I remember testing that simply re-running the same executable resulted in the same test output.
Is it possible that there are a small number of points which access uninitialised memory which changes on each run for these thorns?
#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+---------------------------------------
Comment (by eschnett):
Since the output changes only when the executable changes, it may be that the compiler is generating different code. Certain optimisations may depend on the order in which certain internal compiler data structures are laid out, which may depend e.g. on the time and date.
Can you save two executables? I would like to compare the disassembled output of the routine generating these data, and see whether they differ.
#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+---------------------------------------
Comment (by hinder):
I have removed the "rm" command, so executables will be saved from now on in the automated system. I will make them available when they appear.
#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+---------------------------------------
Comment (by eschnett):
Ian, can you provide this additional feedback now?
#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: hinder Type: defect | Status: accepted Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+--------------------------------------- Changes (by hinder):
* owner: => hinder * status: new => accepted
#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: hinder Type: defect | Status: accepted Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+---------------------------------------
Comment (by hinder):
I have seen something similar to this recently as well. The test GRHydro_test_tov_ppm_ML actually passed on one run and failed when it was run again. The only commit between these two modified unrelated test data and parameter files, and couldn't possibly have changed the behaviour of GRHydro.
The first run was http://git.barrywardell.net/EinsteinToolkitTestResults.git/tree/10647caa473a... and the test passed (http://git.barrywardell.net/EinsteinToolkitTestResults.git/blob/10647caa473a...). The second run was http://git.barrywardell.net/EinsteinToolkitTestResults.git/tree/d7442650e1d7... and the tests failed (http://git.barrywardell.net/EinsteinToolkitTestResults.git/blob/d7442650e1d7...).
The only commit between these two runs was https://github.com/barrywardell/EinsteinExact/commit/26f0b04b7af196673b10043... which could not change the results for the GRHydro test.
Note that this test has been consistently passing when run using the old test system (http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/).
The executables are available in datura:/lustre/datura/ianhin/projects/temp.
#727: Test output changes every time some tests are run ------------------------------------+--------------------------------------- Reporter: hinder | Owner: Type: defect | Status: assigned Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: testsuites QuasiLocalMeasures IDAxiOddBrillBH ------------------------------------+--------------------------------------- Changes (by hinder):
* owner: hinder => * status: accepted => assigned
trac@lists.einsteintoolkit.org
-
Einstein Toolkit