<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Wei and all,<br>
<br>
Thank you for your suggestion. <br>
<br>
The properties.ini file and the two relations you mentioned are very useful to check that what I specify as options when submitting the job is actually what I want.<br>
<br>
Indeed, specifying at the same time e.g.<br>
--procs 32</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
--num-threads 32</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
--ppn-used 32<br>
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I can force the use of 32 cores, overriding the settings in the machine.ini file, without getting inconsistency warnings.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>Therefore</b>, <b>it seems that if I am to leave the <machine>.ini file unchanged, all 3 options above are needed.</b></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
<br>
<b>Now, taking also into account the cluster's billing policy</b> (that Roland Haas flagged in his answer), I am trying to understand how best to formulate my resource requests.<br>
<br>
So, I created a new simulation, not making any specific resource request via the command line, to see what happens with the default values in <machine.ini>, i.e:</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<code>sim create-submit tov_test3 --parfile par/tov_ET.par --walltime 00:20:00</code></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
I have attached the respective properties.ini, .err and .out files.<br>
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>Inspecting the files, do I understand correctly that only 56 threads/cores were used in this scenario?</b></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Given that in the Leonardo cluster there are 112 cores per node (namely 2 x <a title="https://www.intel.com/content/www/us/en/products/sku/231746/intel-xeon-platinum-8480-processor-105m-cache-2-00-ghz/specifications.html" id="LPlnk724104" href="https://www.intel.com/content/www/us/en/products/sku/231746/intel-xeon-platinum-8480-processor-105m-cache-2-00-ghz/specifications.html">
Intel® Xeon® Platinum 8480+ Processor</a> , each of which has 56 cores) does this mean that this way I am using only half the node?<br>
<br>
Should I specify: </div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<code>--procs 112 --num-threads 112</code> </div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
to use one whole node in my case?<br>
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
For reference, I have also attached the ini file that I am using (note that there we have the value
<code>num-threads=56</code>).<br>
Does this make sense in the light of the above discussion?</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
Best regards,<br>
Panayotis</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
<br>
<br>
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div class="elementToProof" id="divRplyFwdMsg">
<div class="elementToProof" style="direction: ltr; font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>From:</b> Wei Sun <wsun3@nd.edu><br>
<b>Sent:</b> Friday, September 26, 2025 4:58 PM<br>
<b>To:</b> IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF@units.it><br>
<b>Cc:</b> users@einsteintoolkit.org <users@einsteintoolkit.org><br>
<b>Subject:</b> Re: Inconsistency warnings: cores/threads mismatch [Leonardo cluster]</div>
<div class="elementToProof" style="direction: ltr;"> </div>
</div>
<p class="elementToProof" style="direction: ltr; margin-top: 1em; margin-bottom: 1em; color: rgb(0, 0, 0);">
Hi Panayotis,</p>
<p class="elementToProof" style="direction: ltr; margin-top: 1em; margin-bottom: 1em; color: rgb(0, 0, 0);">
Can you try specifying <code>--procs</code>, <code>--num-threads</code>, and <code>
--ppn-used</code> at the same time? It looks like your machine defaulted to <code>
--ppn-used=112</code>, which is not what you want, so you may need to overwrite it.</p>
<p class="elementToProof" style="direction: ltr; margin-top: 1em; margin-bottom: 1em; color: rgb(0, 0, 0);">
You can check your allocation details in <code>/output-000x/SIMFACTORY/properties.ini.
</code>A useful strategy is to verify whether the allocation matches these relations:</p>
<pre class="elementToProof"><div class="elementToProof" style="direction: ltr; color: rgb(0, 0, 0);"><code>ppn-used = numthreads * nodeprocs
procs = numthreads * numprocs = nodes * ppn-used
</code></div></pre>
<p class="elementToProof" style="direction: ltr; margin-top: 1em; margin-bottom: 1em; color: rgb(0, 0, 0);">
Below is my understanding (I personally use <code>numsmt=1</code>, so threads = cores in my case). Please correct me if I’m wrong on them:</p>
<div class="elementToProof" style="direction: ltr;"><span style="color: rgb(0, 0, 0);">nodes = number of nodes</span><br>
<span style="color: rgb(0, 0, 0);">procsrequested = total number of cores requested (may not all be used, but you still occupy them on the HPC)</span><br>
<span style="color: rgb(0, 0, 0);">ppn = number of cores per node requested (may not all be used)</span><br>
<span style="color: rgb(0, 0, 0);">numprocs = total number of MPI ranks actually used</span><br>
<span style="color: rgb(0, 0, 0);">nodeprocs = number of MPI ranks per node actually used</span><br>
<span style="color: rgb(0, 0, 0);">procs = total number of cores actually used</span><br>
<span style="color: rgb(0, 0, 0);">numthreads = number of OpenMP threads per MPI rank actually used</span><br>
<span style="color: rgb(0, 0, 0);">ppnused = number of cores per node actually used</span><br>
<span style="color: rgb(0, 0, 0);">numsmt = number of threads per physical core</span></div>
<p class="elementToProof" style="direction: ltr; margin-top: 1em; margin-bottom: 1em; color: rgb(0, 0, 0);">
Best,<br>
Wei</p>
<div class="elementToProof" style="direction: ltr;"><br>
</div>
</body>
</html>