Hello Erik,
thank you for the quick response.
The file "processterminology.rst" in Simfactory is supposed to define these. I see that this file is incomplete -- it does not describe pus, nor does it describe the Simfactory machine database keys. The fact that pus are missing isn't that bad, since no queuing system I know of can allocate individual pus anyway. Also, using hyperthreads looks from a software side the same way as oversubscribing cores.
I see. Thank you for the explanations. I had forgotten about the file. I think I disagree with the "same way as oversubscribing cores." statement (not the idea, just the wording) since it seems as if the OS on stampede2 does report 96 processors (in /proc/cpuinfo's terminology) so a naive application will assume that they are all identical. This is mincing words though :-).
The various misconfigurations and misunderstandings are supposed to be "translated away" by Simfactory. Thus Stampede2's SKX nodes have 48 cores in Simfactory, and if the queueing system requires specifying this as 96, then Simfactory needs to multiply these numbers by 2 when calling sbatch or srun.
ok.
I would try with this description of Stampede2's SKX nodes:
max-num-smt = 2 # the system has hyperthreading num-smt = 1 # ignore hyperthreading by default ppn = 48 # physical cores mpn = 2 # NUMA domains ("sockets")
mpn? I guess is makes sense. Initially I would have used "spn", which I realize is really the number of physical CPU sockets but each socket could have multiple NUMA domains. Note that neither "mpn" or "spn" is used by simfactory for anything (ie "git grep [sm]pn" does only find them in ini files). One of the ini files being etc/syntax/mdb-syntax.ini which states for mpn "memory banks per node" which is a bit ambiguous (could be NUMA domains or could be physical memory sockets) and for spn gives "sockets per node" which is not that much better.
max-num-threads = 96 # using more threads will oversubscribe the pus
How odd, I would have expected this to fail (max-num-treads > ppn) but simfactory happily accepts it and "--procs 96 --num-threads 96 --num-smt 2" gives me a single node while "--procs 96 --num-threads 96 --num-smt 1" requests 2 nodes, which what I wanted.
num-threads = 24 # should probably change this
Yup, this is now 2 in simfactory's master.
I will add these settings to rhaas/stampede2 to see if Jim wants to give that branch a try as well.
Thank you for the explanations.
Yours, Roland