Hi,
I have been looking at why some of AHFinderDirect test cases fail.
First I checked the tests running on 1 processor and found that the cause of the failing test cases was a bug fix made in 2005 where the tests had not been updated afterwards. For those tests I regenerated the output.
One additional test failed by not producing output because it didn't converge to the specified tolerance. I wasn't able to find a version of the code (after the date that the test case had been created) that I was able to compile and that passed the test. For that reason (after discussing with Erik) I disabled that test by renaming the parameter file to someting not ending in .par.
After that I realized that a lot of the tests failed when running on 2 procs. After some investigating, I realized that the cause was to be find in Exact. When going from 1 to 2 procs, the coordinates setup by CoordBase and CartGrid3d can differ at the roundoff level. In the particular case I was looking at in one case the y-coordinate was
-3.3999999999999999
while in the other it was:
-3.4000000000000004
so a difference only in the last digit.
This introduced a difference of the computed 4-metric in the 2 last digits. Now Exact uses 2nd order finite differencing with an epsilon of 1.e-6 to calculate time derivatives of the 3-metric and and the spatial derivatives of the shift that is needed to calculate the extrinsic curvature. In this process there is a significant cancelletion of digits leading to a loss of 6-7 digits so the extrinsic curvature only agreed to 9-10 digits. In other grid points the coordinates were identical and so was the extrinsic curvature. So essentially the process of constructing the extrinsic curvature iteself lead to some arbitrary noise at the 9-10 digit level. When this was fed into AHFinderDirect the differences were amplified even more leading to differences in the mass at the 6th digit level.
Currently Erik is working on improving this (by using higher order finite differencing with a larger epsilon), but the main conclusion is that we have to be very careful when making tests based on initial data from the current version of Exact and may have to reduce the accuracy criteria for pass/fail significantly.
Cheers,
Peter