Comment (by Roland Haas):

If you run on (way) too many MPI ranks then the number of points in a direction can become too small and all kinds of things may break. While one would not like this to happen, it may be hard to avoid in extreme situations.

There are some things that can affect results:

• for hydro run (not usually for pure vacuum runs) checkpoint / recovery is no bitwise identical due to the MoL_PostStep group in POST_RECOVER_VARIABLES which contains a con2prim which is not a projection (ie applying it twice changes results). Can be disabled with a MoL parameter.
• OpenMP reductions (used in all reductions) are not guaranteed to be the same each time they are used due to the order of operations not being fixed and thus the results differ at the roundoff level this affects only output (usually, unless you feed back a reduction result into the simulation eg via using an integral)
• MPI reductions are the same though the MPI standard encourages implementations to return the same answer no matter the number of MPI ranks or the ordering of ranks

There is nothing special about 2^N number of cores from Cactus’s perspective (it’s nice that it only has the single, small, prime factor and this may make domain decomposition a bit easier than say have 68=17*4 cores).

My first suggestion would be to set the MoL parameter that leads to bit identical recovery:

MoL::run_MoL_PostStep_in_Post_Recover_Variables = "no"

‌

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different