#1850: Severe performance problem on Stampede -------------------------+-------------------------------------------------- Reporter: hinder | Owner: Type: defect | Status: confirmed Priority: major | Milestone: Component: SimFactory | Version: development version Resolution: | Keywords: -------------------------+--------------------------------------------------
Comment (by hinder):
The run I originally reported on was run by Seth Hopper. It was using 2 processes per node, each with 8 threads, which is appropriate for Stampede. I believe it was using 96 cores in total (so 6 nodes, 12 processes). We checked the Carpet report of processes and threads, and all was in order. I mentioned that I saw 16 threads in top, because that is the total number of threads; 2 x 8. We were not trying to do anything non-standard. The parameter file was identical to one ran previously on Datura with no problems, but it ran more slowly, which was unexpected. When activating hwloc and SystemTopology, or removing the KMP_AFFINITY line, it went faster by a factor of 8.
I think that SimFactory's machine database should provide reasonable performance by default, and not require people to use hwloc and SystemTopology to avoid an 8-times slowdown. I don't think the current situation is just "suboptimal"; I think it is a bug. What does that KMP_AFFINITY setting do? Do you think it is correct? If it is not feasible to set the affinity properly in simfactory, then I think the best thing is for simfactory to not set it at all, and rely on the system default. The performance may not be optimal, but it shouldn't be 8 times too slow. Then, to get top performance, people can set affinity by activating those thorns (or they can be activated automatically).