[Users] simfactory's num-smt and max-num-smt options and hyperthreading

[Erik Schnetter]

Roland

Unfortunately, the terminology used in general is quite confusing. The fact
that each CPU vendor, HPC system, and queueing software (!) come up with
their own interpretation doesn't help. The fact that some installation
describe their system in the wrong way (presumably to simplify things for
new users) doesn't help.

I know of one place that defines things unambiguously, which is the hwloc
library. It can also examine systems and output their configuration, which
helps. Thus I (and, by implication, Simfactory) uses these definitions.

A "core" is thus a physical core. A "logical core" is called a processing
unit (PU).

Hardware has nodes, sockets, cores, and pus. These are requested by
Simfactory and allocated by the queuing system.

Software uses processes and threads. These are set up via mpirun, OpenMP,
and similar mechanisms.

The file "processterminology.rst" in Simfactory is supposed to define
these. I see that this file is incomplete -- it does not describe pus, nor
does it describe the Simfactory machine database keys. The fact that pus
are missing isn't that bad, since no queuing system I know of can allocate
individual pus anyway. Also, using hyperthreads looks from a software side
the same way as oversubscribing cores.

The various misconfigurations and misunderstandings are supposed to be
"translated away" by Simfactory. Thus Stampede2's SKX nodes have 48 cores
in Simfactory, and if the queueing system requires specifying this as 96,
then Simfactory needs to multiply these numbers by 2 when calling sbatch or
srun.

I would try with this description of Stampede2's SKX nodes:

max-num-smt     = 2   # the system has hyperthreading
num-smt         = 1   # ignore hyperthreading by default
ppn             = 48   # physical cores
mpn             = 2   # NUMA domains ("sockets")
max-num-threads = 96   # using more threads will oversubscribe the pus
num-threads     = 24   # should probably change this
memory          = 196608   # check this
nodes           = 1736   # check this
min-ppn         = 48   # the queueing system always allocates full nodes

-erik



On Wed, Feb 21, 2018 at 1:31 PM, Roland Haas <rhaas at illinois.edu> wrote:

> Hello all,
>
> simfactory's machine.ini files support options num-smt and max-num-smt
> which is take to be related to simultaneous multithreading (ie
> hyper-threadig on Intel CPUs).
>
> However it is not clear to me how to use them properly (looking at
> other .ini files and how simfactory uses them in its simlib.py source
> file) in relation to ppn, num-threads etc.
>
> My initial assumption would have been that ppn is the number of logical
> cores present (ie the number of processors from /proc/cpuinfo) and that
> I'd describe a stampede2-skx compute node (with hyperthreading) as:
>
> ppn = 96
> num-threads = 2 # best according to Jim Healy
> max-num-threads = 96
> max-num-smt = 2
> num-smt = 1
>
> yet with these settings doing a:
>
> sim/bin/submit --procs 96 ...
>
> requests only a single compute node while I would have hoped two
> compute nodes (due to num-smt=1 being the default). I had a look at the
> various computeX.ini files and philip.ini but did not find much that
> helped me in them.
>
> So assuming I would like to do the following:
>
> * describe a stampede2-skx compute node to simfactory
> * so that one can use hypterthreading if desired
> * so that hyperthreading is off by default
>
> how would I do so?
>
> Side note: smt should be described on the ET wiki in the setting up a
> new machine page once this has been settled.
>
> Yours,
> Roland
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>
>


-- 
Erik Schnetter <schnetter at cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20180221/d57be237/attachment.html