Hi,
I recently found OpenMP runs of ET can make different results depending on the number of threads (NT=1 vs. NT neq 1). In some experiments, the difference becomes noticeable only after a long time, but you can see the difference even for the TOV test run with static_tov.par (I compared the time variation of rho_max). With the same parameter setup except for the extended cctk_final time, the difference becomes noticeable around t = 1300. If you take a low order reconstruction such as VanLeerMC2, you can see the difference much earlier. Until t=5000, the differences do not grow drastically and tiny, but non-negligible at all. Things become more serious for more dynamical tests, such as binary neutron star merges.
The difference does not disappear even though I remove all the OpenMP parallellization parts of GRHydro. That means something wrong in other than GRHydro. But one more weird thing is here. The result of No-openmp-GRhydro is not equal to the result of Openmp-GRhydro for the same number of threads. This problem can be fixed by removing the openmp parts in GRHydro_Prim2Con.F90 (I checked only non-MHD routines), even though I couldn't find anything wrong in GRHyro_Prim2Con.F90.
Now, turning to non-GRHydro, I ran QC0 (qc0-mclachlan.par) much longer than the original final_time. Here also I found the floating point calculation is not repeatable even though the difference is tiny; at t=100 which is the original cctk_final_time of qc0-mclachlan.par 100 0.0595683707179757 (for NT=1) 100 0.0595683707137867 (for NT=4)
However, in the private project which uses higher resolution and higher order (8th order) differencing method, I don't see any difference even in the last decimal place at least until t ~ 670. Is it really improved by the higher resolution etc.?
I did the TOV tests on two different machines with Intel Xeon cpus and Intel 11.1 compilers. One is Queenbee and the other is a domestic cluster named Tachyon2. Please take a look at the attached optionlist of Tachyon2. In addition to the default option given in the file, also I tested with; -O2 -no-ip -O3 -no-ip -O3
Any hints or comments would be appreciated.
Cheers,
Hee Il