Comment (by Ken Hui):

For all simulations I did for this test, there is no checkpoint/recovery cycle; and the time-series h+ hx data in mp_psi4 are different in (almost) every line. I mentioned that there are two groups of output:

• (i) number of cores = 8 / 12 / 32 / 36
• (ii) number of cores = 16 / 17 / 24 / 48

what I meant is that, among all simulations in the same group, the mp_psi4 outputs are only different after ~10 decimal places even after a long simulation time; if I compare group (i) and group (ii), for example the mp_psi4 results from 16-core or 32-core, at first they differ at roughly 1 in 10^10, but later on the discrepancy starts growing (see figure, viewed by vim -d).

For the OpenMP part, if I understand correctly, you mean even if I issue

export OMP_NUM_THREADS=1

before the mpirun command every time, there might still problem led by its operation?

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different