[Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

Fri Sep 26 03:53:21 CDT 2025

Panayotis,
   Did you try using --procs=16 --num-threads=1?
I never used --cores and I don't know the difference with procs.

Cheers,
Bruno

Il giorno gio 25 set 2025 alle ore 17:29 IOSIF PANAGIOTIS <
PANAGIOTIS.IOSIF at units.it> ha scritto:

> Hi all,
>
> I am bumping this follow-up question regarding inconsistency warnings when
> requesting resources during a job submit, in case someone has some
> suggestion.
>
> Thanks,
> Panayotis
>
> ------------------------------
> *From:* IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF at units.it>
> *Sent:* Thursday, September 18, 2025 3:25 PM
> *To:* Roland Haas <rhaas at mail.ubc.ca>
> *Cc:* Einstein Toolkit Users <users at einsteintoolkit.org>
> *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch
> [Leonardo cluster]
>
> Dear Roland,
>
> I am following up on my question about how to correctly request the
> resources that I need.
>
> Here is a reminder of the context:
>
>    - I have installed ETK on the Leonardo, DCGP partition at CINECA
>    - I want to run a TOV star example requesting a different number of
>    cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.
>
>
>    -
>
> I use this command to submit the simulation (sim is the typical alias for
> "./simfactory/bin/sim"):
>
>    - sim create-submit tov_test --parfile par/tov_ET.par --cores 16
>    --num-threads 1 --walltime 00:20:00
>
>
> The config files I use are here, and they are based on Bruno Giacomazzo's
> originals:
>
>    - https://github.com/piosif/einstein-toolkit-config-leonardo/
>
>
> I get a *weird error from the CarpetLib thorn*, which complains that:
>
>
>    - *The grid structure is inconsistent.  It is impossible to continue.*
>
>
> I have attached the simulation .err and .out files.
>
> FWIW, inspecting the very beginning of the two files, it seems that the
> overall setup I used (config files and submit command) results in:
>
>
>    - MPI processes: 16
>    - OpenMP threads per core:    0.14285714285714285
>
>
> This is 16 divided by 112 (the total cores in each node), and it does not
> seem to be what I want.
> I would expect that each OpenMP thread should be assigned to one core.
> Furthermore, for that simple test and if my understanding of the basics of
> MPI and OpenMP is correct, it would make more sense to have just one MPI
> process and 16 OpenMP threads.
>
> Could you help me understand what is wrong with my submit command above in
> connection to the options specified in my config files?
>
> Best,
> Panayotis
>
> ------------------------------
> *From:* Roland Haas <rhaas at mail.ubc.ca>
> *Sent:* Wednesday, September 10, 2025 5:59 PM
> *To:* IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF at units.it>
> *Cc:* Einstein Toolkit Users <users at einsteintoolkit.org>
> *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch
> [Leonardo cluster]
>
> Hello Panayotis,
>
> "procs" as used by simfactory is (for historical reasons) a misleading
> term
> (
> https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html
> ).
>
> Really "procs" is the total number of threads that will be started,
> which usually will be the same as total number of (logical) cpus being
> used, and again usually the same as the number of (phyisal) cores used.
>
> If using --cores 16 you are asking for a total of 16 threads to be
> created which should be assigned to 16 cores. You should then select a
> number of threads-per-rank using the `--num-threads` option that is
> good for your setup. For 16 cores most likely `--num-threads 1` (ie one
> thread per MPI rank) is best. This will most likely fix your issue.
>
> Note that this will leave some cores unused and a number of simfactory
> machine description files will not handle that case gracefully since
> they assume that you will always use at least one full node.
>
> For you questions:
>
> * this is usually more of an issue with the SubmitScript and RunScript
>   rather than the ini file and the simfactory command line, though
>   in your case the init script has `num-threads = 56` which sets the
>   default number of for `--num-threads` to 56, which is a bit high (but
>   is overruled by a --num-threads option on your command line)
> * yes, specifying --num-threads on the command line will override the
>   value in the ini file
>
> Yours,
> Roland
>
>
> > [CAUTION: Non-UBC Email]
> >
> > Hi all,
> >
> > After having installed the toolkit on Leonardo, CINECA, I wanted to try
> the TOV examples.
> >
> > However, I am getting the following warnings:
> > Warning: Too many threads per process specified: specified
> num-threads=56 (ppn-used is 112)
> > Warning: Total number of threads and number of threads per process are
> inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer
> multiple of num-threads)
> > Warning: Total number of threads and number of cores per node are
> inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of
> ppn-used)
> > From those, it seems that I am doing something wrong either in my
> machine.ini file or in the resources request when submitting the job.
> >
> > Here is the command I use to run the TOV example:
> > ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16
> --walltime 00:20:00
> >
> > And here is the ini file I am using (the "Cluster characteristics"
> section is identical to Bruno Giacomazzo's file<
> https://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98
> >:
> >
> >
> https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonardo-dcgp1.ini
> >
> > The job ran and completed fine, but from what I understand, it ran on 56
> cores instead of the 16 that I requested.
> >
> > From what I understand, the problem is that I am requesting just 16
> cores for this simple test, but due to the cluster specs in the ini file,
> SimFactory expects me to request a multiple of 56.
> > Is that so?
> >
> > So, my question is the following:
> >
> >   *
> > I want to do some tests, running a single star e.g. on a single node
> using 16, 32, 64 cores (to understand the timescale required). Do I need to
> change my ini file?
> >   *
> > Or is there some option to specify when submitting the job that will
> override the cluster specs?
> >
> > I read the simfactory's terminology explanation on nodes, threads etc<
> https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html>,
> but it is not clear to me how to request the cores I need.
> >
> > Best,
> > Panayotis
>
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>

-- 

Prof. Bruno Giacomazzo
Deputy Director of the Department of Physics
University of Milano-Bicocca
Piazza della Scienza 3
20126 Milano
Italy

email: bruno.giacomazzo at unimib.it
phone: (+39) 02 6448 2321
web: http://www.brunogiacomazzo.org

---------------------------------------------------------------------
There are only 10 types of people in the world:
Those who understand binary, and those who don't
----------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250926/94bfe99d/attachment.htm>