<div dir="ltr">Roland<div><br></div><div>Unfortunately, the terminology used in general is quite confusing. The fact that each CPU vendor, HPC system, and queueing software (!) come up with their own interpretation doesn't help. The fact that some installation describe their system in the wrong way (presumably to simplify things for new users) doesn't help.</div><div><br></div><div>I know of one place that defines things unambiguously, which is the hwloc library. It can also examine systems and output their configuration, which helps. Thus I (and, by implication, Simfactory) uses these definitions.</div><div><br></div><div>A "core" is thus a physical core. A "logical core" is called a processing unit (PU).</div><div><br></div><div>Hardware has nodes, sockets, cores, and pus. These are requested by Simfactory and allocated by the queuing system.</div><div><br></div><div>Software uses processes and threads. These are set up via mpirun, OpenMP, and similar mechanisms.</div><div><br></div><div>The file "processterminology.rst" in Simfactory is supposed to define these. I see that this file is incomplete -- it does not describe pus, nor does it describe the Simfactory machine database keys. The fact that pus are missing isn't that bad, since no queuing system I know of can allocate individual pus anyway. Also, using hyperthreads looks from a software side the same way as oversubscribing cores.</div><div><br></div><div>The various misconfigurations and misunderstandings are supposed to be "translated away" by Simfactory. Thus Stampede2's SKX nodes have 48 cores in Simfactory, and if the queueing system requires specifying this as 96, then Simfactory needs to multiply these numbers by 2 when calling sbatch or srun.</div><div><br></div><div>I would try with this description of Stampede2's SKX nodes:</div><div><br></div><div><div>max-num-smt = 2 # the system has hyperthreading</div><div>num-smt = 1 # ignore hyperthreading by default</div><div>ppn = 48 # physical cores</div><div>mpn = 2 # NUMA domains ("sockets")</div><div>max-num-threads = 96 # using more threads will oversubscribe the pus</div><div>num-threads = 24 # should probably change this</div><div>memory = 196608 # check this</div><div>nodes = 1736 # check this</div><div>min-ppn = 48 # the queueing system always allocates full nodes</div></div><div><br></div><div>-erik</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 21, 2018 at 1:31 PM, Roland Haas <span dir="ltr"><<a href="mailto:rhaas@illinois.edu" target="_blank">rhaas@illinois.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello all,<br>
<br>
simfactory's machine.ini files support options num-smt and max-num-smt<br>
which is take to be related to simultaneous multithreading (ie<br>
hyper-threadig on Intel CPUs).<br>
<br>
However it is not clear to me how to use them properly (looking at<br>
other .ini files and how simfactory uses them in its simlib.py source<br>
file) in relation to ppn, num-threads etc.<br>
<br>
My initial assumption would have been that ppn is the number of logical<br>
cores present (ie the number of processors from /proc/cpuinfo) and that<br>
I'd describe a stampede2-skx compute node (with hyperthreading) as:<br>
<br>
ppn = 96<br>
num-threads = 2 # best according to Jim Healy<br>
max-num-threads = 96<br>
max-num-smt = 2<br>
num-smt = 1<br>
<br>
yet with these settings doing a:<br>
<br>
sim/bin/submit --procs 96 ...<br>
<br>
requests only a single compute node while I would have hoped two<br>
compute nodes (due to num-smt=1 being the default). I had a look at the<br>
various computeX.ini files and philip.ini but did not find much that<br>
helped me in them.<br>
<br>
So assuming I would like to do the following:<br>
<br>
* describe a stampede2-skx compute node to simfactory<br>
* so that one can use hypterthreading if desired<br>
* so that hyperthreading is off by default<br>
<br>
how would I do so?<br>
<br>
Side note: smt should be described on the ET wiki in the setting up a<br>
new machine page once this has been settled.<br>
<br>
Yours,<br>
Roland<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
</font></span><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.<wbr>org/mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div><div><br></div></div></div>
</div>