On 22 Feb 2013, at 17:43, Frank Loeffler <knarf@cct.lsu.edu> wrote:

> Hi Kentaro,
>
> On Fri, Feb 22, 2013 at 04:48:48PM +0100, Kentaro Takami wrote:
>> I encountered the problem in which the simulation result is randomly different
>> at each run( of course I used same executable, par file, HPC and so on.).
>
> I see the same for some of my runs, but I wouldn't expect otherwise.
>
> In my case the simulation depends on 'global' quantities, e.g.,
> quantities obtained by reductions. Reductions cannot be done exactly, at
> least not efficiently. Thus, values obtained via reductions (can) have
> always an error that is different even when running the same simulation
> twice, at least when done in parallel. If your simulation depends on
> this, then these (small) differences can quickly grow, especially in
> iterative schemes, e.g., hydro.

I disagree.  Reductions should be deterministic; assuming the same number of MPI processes, the contributions to the reduction should always be added in the same order.  If you change the number of MPI processes, then I agree that the result of reductions can change, as the order of the sum over points will change, and floating point addition is not associative.