Hi,

I looked into this a bit more today. While I tried to check convergence I noticed that if I increased the resolution on the grid (e.g. from 0.2 to 0.1) I got identical results on 1 and 2 processors. Playing around with the resolution I wasn't able to find another case where I got different results (though this is no guarantee that such cases do not exist). To investigate this further I examined how the irreducible mass depends on the grid resolution, especially in the neighbourhood of 0.2. The attached plot shows the result of that investigation. I plot the irreducible mass as function of abs(dx-0.2). The red curve approaches 0.2 from below (0.19, 0.199 and so on) and the green curve approaches from above (0.21, 0.201 and so on). I have artificially added the results for 1 and 2 processors for dx=0.2 at the x-coordinate 1e-15 in the plot in order to show the magnitude of the difference. I repeat, that in all the cases where the resolution is different from 0.2 I do get the same result on 1 or 2 processors (to within the testsuite tolerance). As you can see there are clear limits when approaching dx=0.2 from below or above. However, the limits are not the same and also differ from the value at dx=0.2. In addition the difference between the limits is larger than the difference between the 1 and 2 processor result at dx=0.2. I must admit that I don't understand why dx=0.2 should be so special compared to all the other resolutions I've tried.

If we want to have some AHFinderDirect testsuites that give identical results on 1 and 2 processors one solution seem to be to simply change the grid resolution in the current tests and regenerate the output.

I have just checked that modifying the resolution of the test:

Kerr-definition-mean-curvature.par

from 0.2 to 0.19 also results in identical results on 1 and 2 processors.

While this seems to solve the testsuite problem, it still leaves open the question as to why these particular tests fail when the resolution is exactly 0.2.

Cheers,

Peter

On Mon, 15 Nov 2010, Erik Schnetter wrote:

On Mon, Nov 15, 2010 at 11:46 AM, Roland Haas roland.haas@physics.gatech.edu wrote:

...

Meeting minutes:

Present were: Frank, Roland, Bruno, Christian, Ian, Erik

Testsuites

• AHFinderDirect/AEILocalInterp: Carpet/mercurial with full grid gives

identical results on 1 or 2 processors, PUGH with bitant grid does not. This points to PUGHInterp to be th culprit. However Ian sees differences when using Multipole and Carpet as well (for 32 processors). This points to the symmetry handling in AEILocalInterp being the culprit.

I have looked at the interpolation coordinates that AHFinderDirect requests from the interpolator, and how they change during the nonlinear elliptic solve. I have used PUGH, and compared runs on 1 and 2 processes.

The first three times the interpolator is called, the coordinates are almost identical, differing by round-off only (1e-16). This round-off error exists only in the domain of the second process. It seems to be cause by the floating-point representation of the lower domain boundary -0.4, which cannot be represented exactly, thus many coordinate values differ by round-off.

When the interpolator is called the fourth time, there are suddenly substantial differences of the order of 1e-7 everywhere in the domain. My best guess this is that the linear solver, which uses an ILU decomposition as preconditioner (I believe) and thus makes some discrete choices depending on its input, yields a different approximation because of the abovementioned floating-point differences.

This is also evident in the output of AHFinderDirect, if one increases the precision. The trial surfaces' Theta value (which depends on the interpolation result) begins to differ significantly in the same iteration in which the average surface radius (which depends on the interpolator input) begins to differ as well. That is, the interpolation result only differs if the interpolator input differs as well. (Note that there are two interpolator calls per iteration; this is used to evaluate the radial dependence of the Jacobian.)

grep 'proc 0/horizon 1' np?/Kerr*.out | awk '{ print $6,$7,$8; }'

1 process:

1 r_grid=1.8888370318372081 ||Theta||=0.58245864115033152 2 r_grid=1.8382954223001324 ||Theta||=0.41707953161957556 3 r_grid=1.7988326341366765 ||Theta||=0.19206874617879019 4 r_grid=1.7970186650813291 ||Theta||=0.020839031289570303 5 r_grid=1.8003073662257811 ||Theta||=0.00049321965075897409 6 r_grid=1.8003202072919120 ||Theta||=5.0676410161959451e-08 7 r_grid=1.8003202078276501 ||Theta||=3.8071282237872595e-13

2 processes:

1 r_grid=1.8888370318372081 ||Theta||=0.5824586411503323 2 r_grid=1.8382954098754503 ||Theta||=0.41707944199389968 3 r_grid=1.7988327249094958 ||Theta||=0.19206838061444106 4 r_grid=1.7970189739199420 ||Theta||=0.020838947150236913 5 r_grid=1.8003076389626189 ||Theta||=0.00049321768953301467 6 r_grid=1.8003204797819821 ||Theta||=5.0679888046606392e-08 7 r_grid=1.8003204803171862 ||Theta||=3.8437655835998896e-13

In conclusion, I could not find any problem with AHFinderDirect, AEILocalInterp, CarpetInterp, or PUGHInterp. It may in principle be good to have more thorough test suites for these, but this would be independent of the horizon finding problem discussed here.

-erik

-- Erik Schnetter schnetter@cct.lsu.edu   http://www.cct.lsu.edu/~eschnett/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users