[ET Trac] [Einstein Toolkit] #1795: PITTNullCode results depend on number of processors

[Einstein Toolkit]

#1795: PITTNullCode results depend on number of processors
------------------------------------+---------------------------------------
  Reporter:  rhaas                  |       Owner:                     
      Type:  defect                 |      Status:  confirmed          
  Priority:  major                  |   Milestone:                     
 Component:  EinsteinToolkit thorn  |     Version:  development version
Resolution:                         |    Keywords:  PITTNullCode       
------------------------------------+---------------------------------------

Comment (by rhaas):

 Thank you for the updated test, it certainly seems to move into the right
 direction. I still get differences though if I run it with 2 or one
 processors (one has to change test.ccl to remove the lenient ATOL and RTOL
 settings) that are of order 1e-11 relative errors.

 {{{
   Test SphericalHarmonicRecon: regression_test
     "PITTNullCode/SphericalHarmonicRecon/test/regression_test.par"

   Issuing mpirun -np 1 /data/rhaas/postdoc/gr/ET_trunk/exe/cactus_sim
 /data/rhaas/postdoc/gr/ET_trunk/arrangements/PITTNullCode/SphericalHarmonicRecon/test/regression_test.par

    Jn_l[0]_2D.asc: substantial differences
       significant differences on 259 (out of 6627) lines
       maximum absolute difference in column 3 is 1.17932885679295e-11
       maximum absolute difference in column 4 is 7.48709427789152e-12
       maximum relative difference in column 3 is 6.80798181095467e-11
       maximum relative difference in column 4 is 3.29043498091389e-11
       (insignificant differences on 1702 lines)
    NewsB[0]_2D.asc: differences below tolerance on 256 lines
    jcn[5]_2D.asc: differences below tolerance on 405 lines
 }}}

 In principle I would have expected multiple processors to give exactly the
 same answer, though if there are really differences in the coordinates
 then I could believe that it is possible for roundoff level differences to
 grow. The test only does 2 steps though (%%cctk_final_time%% is .1 and
 timestep is 0.05) so 4 orders of magnitude (from 1e-15 roundoff to 1e-11)
 seem large.

 Would it be possible to output the coordinates as an hdf5 file (using
 regular HDF5 output) to check if they actually differ?

 Finally if really 4 ghost zones are required then this should really be
 enforced, either by changing the allowed ranges in param.ccl (if always 4
 are required) or through a PARAMCHECK routine (in case the number of ghost
 zones depends on say the finite difference order).

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1795#comment:6>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit