[Users] OpenMP problems in ET

[Frank Loeffler]

On Sat, Jul 16, 2011 at 01:41:06AM +0900, Hee Il Kim wrote:
> I recently found OpenMP runs of ET can make different results depending on
> the number of threads (NT=1 vs. NT neq 1). In some experiments, the
> difference becomes noticeable only after a long time, but you can see the
> difference even for the TOV test run with static_tov.par (I compared the
> time variation of rho_max). With the same parameter setup except for the
> extended cctk_final time, the difference becomes noticeable around t = 1300.

Differences in results are expected when running on different numbers of
mpi processes or openmp threads. How large these differences get depends
on what exactly is done, but the longer a simulation runs the larger the
difference can, in theory, get. This is true even when there is no bug
and everything goes as it should. The challenge is to be sure that
this is indeed the case, and differences are not creeping in because of
some bug.

One of the possibilities to create differences is when the results of
reductions are used within the simulation. Reductions will necessarily
produce (small) differences depending on the number of MPI processes or
openmp threads - because the order in which the reduction is done
differs and creates a different numerical error. This error shouldn't be
all that large. However, if results from this are fed back into the
simulation, these difference might be amplified, especially if iterative
schemes come into play and the number of taken iterations suddenly
changes because of a tiny change in the residuum shifting it above or
below a given tolerance.

One example where tiny differences can have a large impact is when grids
are moved according to the location of, e.g., a neutron star. Assuming
that the stars are tracked by looking for the maximum of some density, a
tiny change at that location might suddenly make a neighbor the maximum,
resulting in a different region being refined, amplifying differences.

All of these differences should vanish when increasing resolution, and
this seems what you also observe. I am sorry that I cannot give a
general answer, but this should suggest that differences are not
necessarily bad - it all depends on how large these differences are,
whether their origin is understood and whether they are reduced when
increasing resolution.

Frank

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20110719/2b1b0a46/attachment.bin