[ET Trac] [Einstein Toolkit] #1361: disable hyperthreading in loopcontrol by default

[Einstein Toolkit]

#1361: disable hyperthreading in loopcontrol by default
-----------------------------------+----------------------------------------
 Reporter:  rhaas                  |       Owner:     
     Type:  defect                 |      Status:  new
 Priority:  major                  |   Milestone:     
Component:  EinsteinToolkit thorn  |     Version:     
 Keywords:  LoopControl            |  
-----------------------------------+----------------------------------------
 when running with both openmp and sufficiently many threads that
 hyperthreading threads are used, many tests using LoopControl (Cartoon,
 RotatingSymmety180, RotatingSymmetry90) fail.

 This can be tracked down to disabling hyperthreading support in
 LoopControl (ie. turning of hyperthreading makes things work).

 In particular on bethe with smt and 8 physical cores:

 The Cartoon/test_cartoon_2.par test shows differences from the recorded
 results when run with 16 threads (but not with 8 threads). If I then go
 ahead and disable OMP in all ML source files but ML_BSSN_enforce *and*
 comment out the #include "loopcontrol.h", then the difference goes away.
 Adding back #include "loopcontrol.h" brings back the error.
 Some further experimenting with LoopControl's options shows that indeed
 the use_smt_threads option is what causes problems. If I turn it off
 things work fine even with a vanilla source tree. Otherwise relative
 differences are on the order 1e-7 and absolute 1e-11 (in
 momx_z_[2][2].xg). Without smt the results are identical to the stored
 values.

 The issue only occurs in combination of OpenMP, vectorization and
 hyperthreading. The issue is independent of the compiler (both intel 13
 and gcc 4.4 show the same behaviour), and vectorization (sse2) and many
 threads (up to 4 times the number of physical cores) works fine on non-smt
 machines.

 I propose to disable LoopControl::use_smt_threads by default. Note that we
 cannot completely remove it since apparently for Vesta (a Blue Gene/Q) smt
 is required to get and multi-threading at all.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1361>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit