On 22 Feb 2013, at 17:43, Frank Loeffler knarf@cct.lsu.edu wrote:
Hi Kentaro,
On Fri, Feb 22, 2013 at 04:48:48PM +0100, Kentaro Takami wrote:
I encountered the problem in which the simulation result is randomly different at each run( of course I used same executable, par file, HPC and so on.).
I see the same for some of my runs, but I wouldn't expect otherwise.
In my case the simulation depends on 'global' quantities, e.g., quantities obtained by reductions. Reductions cannot be done exactly, at least not efficiently. Thus, values obtained via reductions (can) have always an error that is different even when running the same simulation twice, at least when done in parallel. If your simulation depends on this, then these (small) differences can quickly grow, especially in iterative schemes, e.g., hydro.
I disagree. Reductions should be deterministic; assuming the same number of MPI processes, the contributions to the reduction should always be added in the same order. If you change the number of MPI processes, then I agree that the result of reductions can change, as the order of the sum over points will change, and floating point addition is not associative.
The only "excuse" I have seen for the same simulation (exe, parfile, machine, etc) giving different results on different runs is related to a comment in http://software.intel.com/sites/default/files/article/164389/fp-consistency-... (section "Second Example from WRF ") which says that depending on the alignment of the address in memory of a particular loop, either a vectorised or a nonvectorised version of the loop may be called.
Kentaro:
What quantity is different, and by how much? What happens if you compile without optimisation? Are the differences present in the initial data, or just after some evolution steps? How many steps? Where in the grid do the differences appear?
Finally I'm doubting the origin of the problem is "CactusNumerical/Dissipation", because when I don't activate this thorn, the problem can be disappeared.
This is interesting. So, you say you don't see random changes when Dissipation isn't active, but you do when you activate it, and everything else is the same? Which variables do you apply dissipation to?
In this thorn, "dissipation.c" calls the fortran subroutine "apply_dissipation.F77" directly without scheduling in CACTUS. I guess the variables between C and fortran become inconsistency in this stage.
I don't see a problem in the code with this. Calling fortran from C the way done in this thorn seems to be ok.