[Users] OpenMP problems in ET

[Hee Il Kim]

Hi,

I recently found OpenMP runs of ET can make different results depending on
the number of threads (NT=1 vs. NT neq 1). In some experiments, the
difference becomes noticeable only after a long time, but you can see the
difference even for the TOV test run with static_tov.par (I compared the
time variation of rho_max). With the same parameter setup except for the
extended cctk_final time, the difference becomes noticeable around t = 1300.
If you take a low order reconstruction such as VanLeerMC2, you can see the
difference much earlier. Until t=5000, the differences do not grow
drastically and tiny, but non-negligible at all. Things become more serious
for more dynamical tests, such as binary neutron star merges.

The difference does not disappear even though I remove all the OpenMP
parallellization parts of GRHydro. That means something wrong in other than
GRHydro. But one more weird thing is here. The result of No-openmp-GRhydro
is not equal to the result of Openmp-GRhydro for the same number of threads.
This problem can be fixed by removing the openmp parts in
GRHydro_Prim2Con.F90 (I checked only non-MHD routines), even though I
couldn't find anything wrong in GRHyro_Prim2Con.F90.

Now, turning to non-GRHydro, I ran QC0 (qc0-mclachlan.par) much longer than
the original final_time. Here also I found the floating point calculation is
not repeatable even though the difference is tiny; at t=100 which is the
original cctk_final_time of qc0-mclachlan.par
100 0.0595683707179757 (for NT=1)
100 0.0595683707137867 (for NT=4)

However, in the private project which uses higher resolution and higher
order (8th order) differencing method, I don't see any difference even in
the last decimal place at least until t ~ 670. Is it really improved by the
higher resolution etc.?

I did the TOV tests on two different machines with Intel Xeon cpus and Intel
11.1 compilers. One is Queenbee and the other is a domestic cluster named
Tachyon2. Please take a look at the attached optionlist of Tachyon2. In
addition to the default option given in the file, also I tested with;
-O2 -no-ip
-O3 -no-ip
-O3

Any hints or comments would be appreciated.

Cheers,

Hee Il
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20110716/458d149b/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tachyon2.cfg
Type: application/octet-stream
Size: 3882 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20110716/458d149b/attachment.obj