[ET Trac] [Einstein Toolkit] #2096: SphericalHarmonicRecon and SphericalHarmonicReconGen tests fail on Jenkins build machine

[Einstein Toolkit]

#2096: SphericalHarmonicRecon and SphericalHarmonicReconGen tests fail on Jenkins
build machine
-----------------------------------+----------------------------------------
 Reporter:  hinder                 |       Owner:                     
     Type:  defect                 |      Status:  new                
 Priority:  major                  |   Milestone:  ET_2018_02         
Component:  EinsteinToolkit thorn  |     Version:  development version
 Keywords:                         |  
-----------------------------------+----------------------------------------
 (Adapted from
 http://lists.einsteintoolkit.org/pipermail/users/2017-July/005671.html)

 The following three tests fail on the Jenkins build machine:

 SphericalHarmonicRecon.regression_test/2procs
 SphericalHarmonicReconGen.SpEC-dat-test/2procs
 SphericalHarmonicReconGen.SpEC-h5-test/2procs

 These tests all pass on one process but fail on two processes, using the
 ubuntu.cfg optionlist.  They all seem to pass on multiple processes on all
 other machines, including my laptop with gcc.

 The first, SphericalHarmonicRecon.regression_test, fails like this:

 {{{
 WARNING level 0 from host 7ce14e5707a0 process 0
   while executing schedule bin NullEvol_Initial, routine
 NullEvolve::NullEvol_InitialSlice
   in thorn NullEvolve, file NullEvol_InitialSlice.F90:42:
   ->  Error
 }}}

 The second, SphericalHarmonicReconGen.SpEC-dat-test, fails like this:

 {{{
 NewsB_scri.L02Mm01.asc: substantial differences
       significant differences on 1 (out of 2) lines
       maximum absolute difference in column 1 is 963
       maximum absolute difference in column 2 is 0.000185770963653907
       maximum absolute difference in column 3 is 0.000142466608463344
       maximum relative difference in column 1 is 1
       maximum relative difference in column 2 is 1
       maximum relative difference in column 3 is 1
       ...
 }}}

 The third, SphericalHarmonicReconGen.SpEC-h5-test, fails like this:

 {{{
 NewsB_scri.L02Mm01.asc: substantial differences
       significant differences on 1 (out of 2) lines
       maximum absolute difference in column 1 is 963
       maximum absolute difference in column 2 is 0.000185770963653907
       maximum absolute difference in column 3 is 0.000142466608463344
       maximum relative difference in column 1 is 1
       maximum relative difference in column 2 is 1
       maximum relative difference in column 3 is 1
       ...
 }}}

 I suspect the second and third failures have the same cause.  These tests
 don't seem to fail on any other machines.

 See the discussion thread
 http://lists.einsteintoolkit.org/pipermail/users/2017-July/thread.html#5671
 for more information.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/2096>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit