[Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

[IOSIF PANAGIOTIS]

Hi Wei and all,

Thank you for your suggestion.

The properties.ini file and the two relations you mentioned are very useful to check that what I specify as options when submitting the job is actually what I want.

Indeed, specifying at the same time e.g.
--procs 32
--num-threads 32
--ppn-used 32

I can force the use of 32 cores, overriding the settings in the machine.ini file, without getting inconsistency warnings.

Therefore, it seems that if I am to leave the <machine>.ini file unchanged, all 3 options above are needed.


Now, taking also into account the cluster's billing policy (that Roland Haas flagged in his answer), I am trying to understand how best to formulate my resource requests.

So, I created a new simulation, not making any specific resource request via the command line, to see what happens with the default values in <machine.ini>, i.e:
sim create-submit tov_test3 --parfile par/tov_ET.par --walltime 00:20:00

I have attached the respective properties.ini, .err and .out files.

Inspecting the files, do I understand correctly that only 56 threads/cores were used in this scenario?

Given that in the Leonardo cluster there are 112 cores per node (namely 2 x Intel® Xeon® Platinum 8480+ Processor<https://www.intel.com/content/www/us/en/products/sku/231746/intel-xeon-platinum-8480-processor-105m-cache-2-00-ghz/specifications.html> , each of which has 56 cores) does this mean that this way I am using only half the node?

Should I specify:
--procs 112 --num-threads 112
to use one whole node in my case?

For reference, I have also attached the ini file that I am using (note that there we have the value num-threads=56).
Does this make sense in the light of the above discussion?

Best regards,
Panayotis






________________________________
From: Wei Sun <wsun3 at nd.edu>
Sent: Friday, September 26, 2025 4:58 PM
To: IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF at units.it>
Cc: users at einsteintoolkit.org <users at einsteintoolkit.org>
Subject: Re: Inconsistency warnings: cores/threads mismatch [Leonardo cluster]


Hi Panayotis,

Can you try specifying --procs, --num-threads, and --ppn-used at the same time? It looks like your machine defaulted to --ppn-used=112, which is not what you want, so you may need to overwrite it.

You can check your allocation details in /output-000x/SIMFACTORY/properties.ini. A useful strategy is to verify whether the allocation matches these relations:

ppn-used   = numthreads * nodeprocs
procs      = numthreads * numprocs = nodes * ppn-used


Below is my understanding (I personally use numsmt=1, so threads = cores in my case). Please correct me if I’m wrong on them:

nodes = number of nodes
procsrequested = total number of cores requested (may not all be used, but you still occupy them on the HPC)
ppn = number of cores per node requested (may not all be used)
numprocs = total number of MPI ranks actually used
nodeprocs = number of MPI ranks per node actually used
procs = total number of cores actually used
numthreads = number of OpenMP threads per MPI rank actually used
ppnused = number of cores per node actually used
numsmt = number of threads per physical core

Best,
Wei

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250929/b2972b44/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: properties_tov_test3.ini
Type: application/octet-stream
Size: 1462 bytes
Desc: properties_tov_test3.ini
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250929/b2972b44/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tov_test3.out
Type: application/octet-stream
Size: 150035 bytes
Desc: tov_test3.out
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250929/b2972b44/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tov_test3.err
Type: application/octet-stream
Size: 2308 bytes
Desc: tov_test3.err
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250929/b2972b44/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: leonardo-dcgp1.ini
Type: application/octet-stream
Size: 6708 bytes
Desc: leonardo-dcgp1.ini
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250929/b2972b44/attachment-0007.obj>