[Users] bug in Dissipation thorn?

[Ian Hinder]

On 22 Feb 2013, at 18:13, Erik Schnetter <schnetter at cct.lsu.edu> wrote:

> On Fri, Feb 22, 2013 at 12:01 PM, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
> 
> On 22 Feb 2013, at 17:43, Frank Loeffler <knarf at cct.lsu.edu> wrote:
> 
> > Hi Kentaro,
> >
> > On Fri, Feb 22, 2013 at 04:48:48PM +0100, Kentaro Takami wrote:
> >> I encountered the problem in which the simulation result is randomly different
> >> at each run( of course I used same executable, par file, HPC and so on.).
> >
> > I see the same for some of my runs, but I wouldn't expect otherwise.
> >
> > In my case the simulation depends on 'global' quantities, e.g.,
> > quantities obtained by reductions. Reductions cannot be done exactly, at
> > least not efficiently. Thus, values obtained via reductions (can) have
> > always an error that is different even when running the same simulation
> > twice, at least when done in parallel. If your simulation depends on
> > this, then these (small) differences can quickly grow, especially in
> > iterative schemes, e.g., hydro.
> 
> I disagree.  Reductions should be deterministic; assuming the same number of MPI processes, the contributions to the reduction should always be added in the same order.  If you change the number of MPI processes, then I agree that the result of reductions can change, as the order of the sum over points will change, and floating point addition is not associative.
> 
> MPI assumes that reduction operations are associative.

Ouch.  You are right:

http://www.mpi-forum.org/docs/mpi-11-html/node77.html
> The operation op is always assumed to be associative. All predefined operations are also assumed to be commutative. Users may define operations that are assumed to be associative, but not commutative. The ``canonical'' evaluation order of a reduction is determined by the ranks of the processes in the group. However, the implementation can take advantage of associativity, or associativity and commutativity in order to change the order of evaluation. This may change the result of the reduction for operations that are not strictly associative and commutative, such as floating point addition.
> 
> 
> [] Advice to implementors.
> 
> It is strongly recommended that MPI_REDUCE be implemented so that the same result be obtained whenever the function is applied on the same arguments, appearing in the same order. Note that this may prevent optimizations that take advantage of the physical location of processors. ( End of advice to implementors.) 
> The datatype argument of MPI_REDUCE must be compatible with op. Predefined operators work only with the MPI types listed in Sec. Predefined reduce operations and Sec. MINLOC and MAXLOC . User-defined operators may operate on general, derived datatypes. In this case, each argument that the reduce operation is applied to is one element described by such a datatype, which may contain several basic values. This is further explained in Section User-Defined Operations .
> 
> 

So MPI implementors are strongly recommended to ensure that the result is deterministic, but the standard allows them to deviate from this.

> So does OpenMP, and Carpet. So does SimFactory it its option settings. So does Kranc when it generates code…

Kranc assumes associativity because it simplifies expressions.  This is no different to what the compiler does when it optimises.  So if I change the simplification settings in Kranc, or possibly use a newer version of Mathematica, it may generate different code, just like using a different compiler might.  But at least with the same original source files and tools, I should be able to do the same experiment more than once and get the same results.  If the *runtime* results are allowed to differ, things become very hard to reason about.

-- 
Ian Hinder
http://numrel.aei.mpg.de/people/hinder

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20130222/cf03ed10/attachment.html