Roland

Unfortunately, the terminology used in general is quite confusing. The fact that each CPU vendor, HPC system, and queueing software (!) come up with their own interpretation doesn't help. The fact that some installation describe their system in the wrong way (presumably to simplify things for new users) doesn't help.

I know of one place that defines things unambiguously, which is the hwloc library. It can also examine systems and output their configuration, which helps. Thus I (and, by implication, Simfactory) uses these definitions.

A "core" is thus a physical core. A "logical core" is called a processing unit (PU).

Hardware has nodes, sockets, cores, and pus. These are requested by Simfactory and allocated by the queuing system.

Software uses processes and threads. These are set up via mpirun, OpenMP, and similar mechanisms.

The file "processterminology.rst" in Simfactory is supposed to define these. I see that this file is incomplete -- it does not describe pus, nor does it describe the Simfactory machine database keys. The fact that pus are missing isn't that bad, since no queuing system I know of can allocate individual pus anyway. Also, using hyperthreads looks from a software side the same way as oversubscribing cores.

The various misconfigurations and misunderstandings are supposed to be "translated away" by Simfactory. Thus Stampede2's SKX nodes have 48 cores in Simfactory, and if the queueing system requires specifying this as 96, then Simfactory needs to multiply these numbers by 2 when calling sbatch or srun.

I would try with this description of Stampede2's SKX nodes:

max-num-smt     = 2   # the system has hyperthreading
num-smt         = 1   # ignore hyperthreading by default
ppn             = 48   # physical cores
mpn             = 2   # NUMA domains ("sockets")
max-num-threads = 96   # using more threads will oversubscribe the pus
num-threads     = 24   # should probably change this
memory          = 196608   # check this
nodes           = 1736   # check this
min-ppn         = 48   # the queueing system always allocates full nodes

-erik



On Wed, Feb 21, 2018 at 1:31 PM, Roland Haas <rhaas@illinois.edu> wrote:
Hello all,

simfactory's machine.ini files support options num-smt and max-num-smt
which is take to be related to simultaneous multithreading (ie
hyper-threadig on Intel CPUs).

However it is not clear to me how to use them properly (looking at
other .ini files and how simfactory uses them in its simlib.py source
file) in relation to ppn, num-threads etc.

My initial assumption would have been that ppn is the number of logical
cores present (ie the number of processors from /proc/cpuinfo) and that
I'd describe a stampede2-skx compute node (with hyperthreading) as:

ppn = 96
num-threads = 2 # best according to Jim Healy
max-num-threads = 96
max-num-smt = 2
num-smt = 1

yet with these settings doing a:

sim/bin/submit --procs 96 ...

requests only a single compute node while I would have hoped two
compute nodes (due to num-smt=1 being the default). I had a look at the
various computeX.ini files and philip.ini but did not find much that
helped me in them.

So assuming I would like to do the following:

* describe a stampede2-skx compute node to simfactory
* so that one can use hypterthreading if desired
* so that hyperthreading is off by default

how would I do so?

Side note: smt should be described on the ET wiki in the setting up a
new machine page once this has been settled.

Yours,
Roland

--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .

_______________________________________________
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users




--
Erik Schnetter <schnetter@cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/