[ET Trac] [Einstein Toolkit] #1850: Severe performance problem on Stampede

[Einstein Toolkit]

#1850: Severe performance problem on Stampede
-------------------------+--------------------------------------------------
  Reporter:  hinder      |       Owner:                     
      Type:  defect      |      Status:  confirmed          
  Priority:  major       |   Milestone:                     
 Component:  SimFactory  |     Version:  development version
Resolution:              |    Keywords:                     
-------------------------+--------------------------------------------------

Comment (by hinder):

 The run I originally reported on was run by Seth Hopper.  It was using 2
 processes per node, each with 8 threads, which is appropriate for
 Stampede.  I believe it was using 96 cores in total (so 6 nodes, 12
 processes).  We checked the Carpet report of processes and threads, and
 all was in order.  I mentioned that I saw 16 threads in top, because that
 is the total number of threads; 2 x 8.  We were not trying to do anything
 non-standard.  The parameter file was identical to one ran previously on
 Datura with no problems, but it ran more slowly, which was unexpected.
 When activating hwloc and SystemTopology, or removing the KMP_AFFINITY
 line, it went faster by a factor of 8.

 I think that SimFactory's machine database should provide reasonable
 performance by default, and not require people to use hwloc and
 SystemTopology to avoid an 8-times slowdown.  I don't think the current
 situation is just "suboptimal"; I think it is a bug.  What does that
 KMP_AFFINITY setting do?  Do you think it is correct?  If it is not
 feasible to set the affinity properly in simfactory, then I think the best
 thing is for simfactory to not set it at all, and rely on the system
 default.  The performance may not be optimal, but it shouldn't be 8 times
 too slow.  Then, to get top performance, people can set affinity by
 activating those thorns (or they can be activated automatically).

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1850#comment:14>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit