#1795: PITTNullCode results depend on number of processors ------------------------------------+--------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: confirmed Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: PITTNullCode ------------------------------------+---------------------------------------
Comment (by rhaas):
Thank you for the updated test, it certainly seems to move into the right direction. I still get differences though if I run it with 2 or one processors (one has to change test.ccl to remove the lenient ATOL and RTOL settings) that are of order 1e-11 relative errors.
{{{ Test SphericalHarmonicRecon: regression_test "PITTNullCode/SphericalHarmonicRecon/test/regression_test.par"
Issuing mpirun -np 1 /data/rhaas/postdoc/gr/ET_trunk/exe/cactus_sim /data/rhaas/postdoc/gr/ET_trunk/arrangements/PITTNullCode/SphericalHarmonicRecon/test/regression_test.par
Jn_l[0]_2D.asc: substantial differences significant differences on 259 (out of 6627) lines maximum absolute difference in column 3 is 1.17932885679295e-11 maximum absolute difference in column 4 is 7.48709427789152e-12 maximum relative difference in column 3 is 6.80798181095467e-11 maximum relative difference in column 4 is 3.29043498091389e-11 (insignificant differences on 1702 lines) NewsB[0]_2D.asc: differences below tolerance on 256 lines jcn[5]_2D.asc: differences below tolerance on 405 lines }}}
In principle I would have expected multiple processors to give exactly the same answer, though if there are really differences in the coordinates then I could believe that it is possible for roundoff level differences to grow. The test only does 2 steps though (%%cctk_final_time%% is .1 and timestep is 0.05) so 4 orders of magnitude (from 1e-15 roundoff to 1e-11) seem large.
Would it be possible to output the coordinates as an hdf5 file (using regular HDF5 output) to check if they actually differ?
Finally if really 4 ghost zones are required then this should really be enforced, either by changing the allowed ranges in param.ccl (if always 4 are required) or through a PARAMCHECK routine (in case the number of ghost zones depends on say the finite difference order).