On Sat, Jul 16, 2011 at 01:41:06AM +0900, Hee Il Kim wrote:
I recently found OpenMP runs of ET can make different results depending on the number of threads (NT=1 vs. NT neq 1). In some experiments, the difference becomes noticeable only after a long time, but you can see the difference even for the TOV test run with static_tov.par (I compared the time variation of rho_max). With the same parameter setup except for the extended cctk_final time, the difference becomes noticeable around t = 1300.
Differences in results are expected when running on different numbers of mpi processes or openmp threads. How large these differences get depends on what exactly is done, but the longer a simulation runs the larger the difference can, in theory, get. This is true even when there is no bug and everything goes as it should. The challenge is to be sure that this is indeed the case, and differences are not creeping in because of some bug.
One of the possibilities to create differences is when the results of reductions are used within the simulation. Reductions will necessarily produce (small) differences depending on the number of MPI processes or openmp threads - because the order in which the reduction is done differs and creates a different numerical error. This error shouldn't be all that large. However, if results from this are fed back into the simulation, these difference might be amplified, especially if iterative schemes come into play and the number of taken iterations suddenly changes because of a tiny change in the residuum shifting it above or below a given tolerance.
One example where tiny differences can have a large impact is when grids are moved according to the location of, e.g., a neutron star. Assuming that the stars are tracked by looking for the maximum of some density, a tiny change at that location might suddenly make a neighbor the maximum, resulting in a different region being refined, amplifying differences.
All of these differences should vanish when increasing resolution, and this seems what you also observe. I am sorry that I cannot give a general answer, but this should suggest that differences are not necessarily bad - it all depends on how large these differences are, whether their origin is understood and whether they are reduced when increasing resolution.
Frank