[ET Trac] [Einstein Toolkit] #2200: some test may be non-deterministic

[Einstein Toolkit]

#2200: some test may be non-deterministic
---------------------------------+--------------------
 Reporter:  Roland Haas          |       Type:  defect
   Status:  new                  |   Priority:  unset
Milestone:                       |  Component:  Other
  Version:  development version  |   Keywords:
---------------------------------+--------------------
 For the ET_2018_09 release it seems that tests run on Sep 6th and Sept
 22nd differ and these tests show a different number of failed tests:

 * comet__1_24.log
 * cori__1_16.log
 * osx-homebrew__2_2.log
 * osx-macports__2_2.log

 and given that there should not have been a change in code between those
 days this seems suspicious.

 The particular tests that changed are:

 {{{
 +++ b/results/comet__1_24.log
 @@ -1434,7 +1434,7 @@
        AHFinderDirect: misner1.2-025
           Success: 55 files compared, 9 differ in the last digits
        AHFinderDirect: recoverML-EE
 -         Success: 5 files compared, 5 differ in the last digits
 +         Failure: 2 files missing, 3 files compared, 3 differ, 3 differ
 significantly
        Carpet: 64k2
           Success: 0 files identical
        Carpet: test_restrict_sync
 }}}

 {{{
 +++ b/results/cori__1_16.log
 @@ -1446,7 +1446,7 @@
        CarpetIOHDF5: CarpetWaveToyNewRecover_test_1proc
           Success: 12 files compared, 7 differ in the last digits
        CarpetIOHDF5: CarpetWaveToyRecover_test_1proc
 -         Failure: 12 files missing, 0 files compared, 0 differ
 +         Success: 12 files compared, 7 differ in the last digits
        CarpetIOHDF5: CarpetWaveToyRecover_test_newcp_1proc
           Success: 12 files compared, 7 differ in the last digits
        CarpetIOHDF5: newsep
 }}}

 {{{
 +++ b/results/osx-homebrew__2_2.log
 @@ -1631,7 +1631,7 @@
        PeriodicCarpet: testperiodicinterp
           Success: 1 files identical
        QuasiLocalMeasures: qlm-bl
 -         Failure: 54 files missing, 115 files compared, 115 differ
 +         Success: 169 files compared, 143 differ in the last digits
        QuasiLocalMeasures: qlm-ks
           Success: 169 files compared, 138 differ in the last digits
        QuasiLocalMeasures: qlm-ks-EE
 }}}

 {{{
 +++ b/results/osx-macports__2_2.log
 @@ -1631,7 +1631,7 @@
        PeriodicCarpet: testperiodicinterp
           Success: 1 files identical
        QuasiLocalMeasures: qlm-bl
 -         Failure: 54 files missing, 115 files compared, 115 differ
 +         Success: 169 files compared, 144 differ in the last digits
        QuasiLocalMeasures: qlm-ks
           Success: 169 files compared, 136 differ in the last digits
        QuasiLocalMeasures: qlm-ks-EE
 }}}

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/2200>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit