Comment (by Roland Haas):

Setting

export OMP_NUM_THREADS=1

would at least remove the known and documented source of differences if you have no checkpointing / recovery. With a single thread then I would expect that your results should exactly identical no matter how many MPI ranks. If they are not identical then that could indicate a bug in the ET (or a bug in my understanding of the ET).

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different