[Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

[IOSIF PANAGIOTIS]

Dear Roland,

I am following up on my question about how to correctly request the resources that I need.

Here is a reminder of the context:

  *
I have installed ETK on the Leonardo, DCGP partition at CINECA
  *
I want to run a TOV star example requesting a different number of cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.

  *


I use this command to submit the simulation (sim is the typical alias for "./simfactory/bin/sim"):

  *
sim create-submit tov_test --parfile par/tov_ET.par --cores 16 --num-threads 1 --walltime 00:20:00

The config files I use are here, and they are based on Bruno Giacomazzo's originals:

  *
https://github.com/piosif/einstein-toolkit-config-leonardo/

I get a weird error from the CarpetLib thorn, which complains that:


  *
The grid structure is inconsistent.  It is impossible to continue.

I have attached the simulation .err and .out files.

FWIW, inspecting the very beginning of the two files, it seems that the overall setup I used (config files and submit command) results in:


  *
MPI processes: 16
  *
OpenMP threads per core:    0.14285714285714285

This is 16 divided by 112 (the total cores in each node), and it does not seem to be what I want.
I would expect that each OpenMP thread should be assigned to one core.
Furthermore, for that simple test and if my understanding of the basics of MPI and OpenMP is correct, it would make more sense to have just one MPI process and 16 OpenMP threads.

Could you help me understand what is wrong with my submit command above in connection to the options specified in my config files?

Best,
Panayotis

________________________________
From: Roland Haas <rhaas at mail.ubc.ca>
Sent: Wednesday, September 10, 2025 5:59 PM
To: IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF at units.it>
Cc: Einstein Toolkit Users <users at einsteintoolkit.org>
Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

Hello Panayotis,

"procs" as used by simfactory is (for historical reasons) a misleading
term
(https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html).

Really "procs" is the total number of threads that will be started,
which usually will be the same as total number of (logical) cpus being
used, and again usually the same as the number of (phyisal) cores used.

If using --cores 16 you are asking for a total of 16 threads to be
created which should be assigned to 16 cores. You should then select a
number of threads-per-rank using the `--num-threads` option that is
good for your setup. For 16 cores most likely `--num-threads 1` (ie one
thread per MPI rank) is best. This will most likely fix your issue.

Note that this will leave some cores unused and a number of simfactory
machine description files will not handle that case gracefully since
they assume that you will always use at least one full node.

For you questions:

* this is usually more of an issue with the SubmitScript and RunScript
  rather than the ini file and the simfactory command line, though
  in your case the init script has `num-threads = 56` which sets the
  default number of for `--num-threads` to 56, which is a bit high (but
  is overruled by a --num-threads option on your command line)
* yes, specifying --num-threads on the command line will override the
  value in the ini file

Yours,
Roland


> [CAUTION: Non-UBC Email]
>
> Hi all,
>
> After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.
>
> However, I am getting the following warnings:
> Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112)
> Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads)
> Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used)
> From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.
>
> Here is the command I use to run the TOV example:
> ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00
>
> And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's file<https://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98>:
>
> https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonardo-dcgp1.ini
>
> The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.
>
> From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56.
> Is that so?
>
> So, my question is the following:
>
>   *
> I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?
>   *
> Or is there some option to specify when submitting the job that will override the cluster specs?
>
> I read the simfactory's terminology explanation on nodes, threads etc<https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html>, but it is not clear to me how to request the cores I need.
>
> Best,
> Panayotis


--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250918/2cb33c70/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tov_test.out
Type: application/octet-stream
Size: 144874 bytes
Desc: tov_test.out
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250918/2cb33c70/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tov_test.err
Type: application/octet-stream
Size: 75260 bytes
Desc: tov_test.err
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250918/2cb33c70/attachment-0003.obj>