[Users] bug in Dissipation thorn?

[Ian Hinder]

On 22 Feb 2013, at 17:43, Frank Loeffler <knarf at cct.lsu.edu> wrote:

> Hi Kentaro,
> 
> On Fri, Feb 22, 2013 at 04:48:48PM +0100, Kentaro Takami wrote:
>> I encountered the problem in which the simulation result is randomly different
>> at each run( of course I used same executable, par file, HPC and so on.).
> 
> I see the same for some of my runs, but I wouldn't expect otherwise.
> 
> In my case the simulation depends on 'global' quantities, e.g.,
> quantities obtained by reductions. Reductions cannot be done exactly, at
> least not efficiently. Thus, values obtained via reductions (can) have
> always an error that is different even when running the same simulation
> twice, at least when done in parallel. If your simulation depends on
> this, then these (small) differences can quickly grow, especially in
> iterative schemes, e.g., hydro.

I disagree.  Reductions should be deterministic; assuming the same number of MPI processes, the contributions to the reduction should always be added in the same order.  If you change the number of MPI processes, then I agree that the result of reductions can change, as the order of the sum over points will change, and floating point addition is not associative.

The only "excuse" I have seen for the same simulation (exe, parfile, machine, etc) giving different results on different runs is related to a comment in http://software.intel.com/sites/default/files/article/164389/fp-consistency-102511.pdf (section "Second Example from WRF
 ") which says that depending on the alignment of the address in memory of a particular loop, either a vectorised or a nonvectorised version of the loop may be called.  

Kentaro:

What quantity is different, and by how much?  What happens if you compile without optimisation?  Are the differences present in the initial data, or just after some evolution steps?  How many steps?  Where in the grid do the differences appear?

>> Finally I'm doubting the origin of the problem is "CactusNumerical/Dissipation",
>> because when I don't activate this thorn, the problem can be disappeared.
> 
> This is interesting. So, you say you don't see random changes when
> Dissipation isn't active, but you do when you activate it, and
> everything else is the same? Which variables do you apply dissipation
> to?
> 
>> In this thorn, "dissipation.c" calls the fortran subroutine
>> "apply_dissipation.F77" directly
>> without scheduling in CACTUS.
>> I guess the variables between C and fortran become inconsistency in this stage.
> 
> I don't see a problem in the code with this. Calling fortran from C the
> way done in this thorn seems to be ok.

-- 
Ian Hinder
http://numrel.aei.mpg.de/people/hinder

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20130222/dc9d6cbe/attachment.bin