#1850: Severe performance problem on Stampede -------------------------+-------------------------------------------------- Reporter: hinder | Owner: Type: defect | Status: confirmed Priority: major | Milestone: Component: SimFactory | Version: development version Resolution: | Keywords: -------------------------+--------------------------------------------------
Comment (by hinder):
The tacc_affinity script is recommended for stampede (https://portal.xsede.org/tacc-stampede). It is supposed to guarantee that the processes are distributed among sockets, as well as the memory they allocate. We already use this in the simfactory run script. This is the only place that this can be done, because once Cactus has started, MPI has already initialised itself, and may have allocated memory on the wrong socket if the process has not yet been pinned to a socket.
Erik has indicated that the KMP_AFFINITY setting is likely only correct when you have only a single process per node, and using this variable cannot be correct when you have more than one process per node, because the compiler, which interprets this variable, does not know about the additional processes. I observed that removing the setting of this variable eliminated the performance problem that I saw when not using SystemTopology. I therefore propose that the setting of this variable is removed from the run script.