<div dir="ltr">Panayotis,<br><div> Did you try using --procs=16 --num-threads=1?</div><div>I never used --cores and I don't know the difference with procs.</div><div><br></div><div>Cheers,</div><div>Bruno</div><div><br></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">Il giorno gio 25 set 2025 alle ore 17:29 IOSIF PANAGIOTIS <<a href="mailto:PANAGIOTIS.IOSIF@units.it">PANAGIOTIS.IOSIF@units.it</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1333107120662898781">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi all,<br>
<br>
I am bumping this follow-up question regarding inconsistency warnings when requesting resources during a job submit, in case someone has some suggestion.<br>
<br>
Thanks,<br>
Panayotis</div>
<div id="m_-1333107120662898781appendonsend"></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-1333107120662898781divRplyFwdMsg">
<div style="direction:ltr;font-family:Calibri,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<b>From:</b> IOSIF PANAGIOTIS <<a href="mailto:PANAGIOTIS.IOSIF@units.it" target="_blank">PANAGIOTIS.IOSIF@units.it</a>><br>
<b>Sent:</b> Thursday, September 18, 2025 3:25 PM<br>
<b>To:</b> Roland Haas <<a href="mailto:rhaas@mail.ubc.ca" target="_blank">rhaas@mail.ubc.ca</a>><br>
<b>Cc:</b> Einstein Toolkit Users <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]</div>
<div style="direction:ltr"> </div>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Dear Roland,<br>
<br>
I am following up on my question about how to correctly request the resources that I need.<br>
<br>
Here is a reminder of the context:</div>
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px;list-style-type:disc">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr">I have installed ETK on the Leonardo, DCGP partition at CINECA</div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr">I want to run a TOV star example requesting a different number of cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.</div>
</li></ul>
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);display:block;list-style-type:"- "">
<div role="presentation" style="direction:ltr"><br>
</div>
</li></ul>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I use this command to submit the simulation (sim is the typical alias for "./simfactory/bin/sim"):</div>
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px;list-style-type:disc">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr"><code>sim create-submit tov_test --parfile par/tov_ET.par --cores 16 --num-threads 1 --walltime 00:20:00</code></div>
</li></ul>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
The config files I use are here, and they are based on Bruno Giacomazzo's originals:</div>
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px;list-style-type:disc">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr"><a id="m_-1333107120662898781OWAcaafbeeb-42b6-f976-d0d8-c2efce8a8ba5" href="https://github.com/piosif/einstein-toolkit-config-leonardo/" target="_blank">https://github.com/piosif/einstein-toolkit-config-leonardo/</a></div>
</li></ul>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I get a <b>weird error from the CarpetLib thorn</b>, which complains that:</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px;list-style-type:disc">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr"><i>The grid structure is inconsistent. It is impossible to continue.</i></div>
</li></ul>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<i><br>
</i></div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I have attached the simulation .err and .out files.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
FWIW, inspecting the very beginning of the two files, it seems that the overall setup I used (config files and submit command) results in:</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px;list-style-type:disc">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr">MPI processes: 16</div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div role="presentation" style="direction:ltr">OpenMP threads per core: 0.14285714285714285</div>
</li></ul>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
This is 16 divided by 112 (the total cores in each node), and it does not seem to be what I want.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I would expect that each OpenMP thread should be assigned to one core.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Furthermore, for that simple test and if my understanding of the basics of MPI and OpenMP is correct, it would make more sense to have just one MPI process and 16 OpenMP threads. </div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Could you help me understand what is wrong with my submit command above in connection to the options specified in my config files? <br>
<br>
Best,<br>
Panayotis<br>
<br>
</div>
<div id="m_-1333107120662898781x_appendonsend"></div>
<hr style="direction:ltr;display:inline-block;width:98%">
<div id="m_-1333107120662898781x_divRplyFwdMsg">
<div style="direction:ltr;font-family:Calibri,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<b>From:</b> Roland Haas <<a href="mailto:rhaas@mail.ubc.ca" target="_blank">rhaas@mail.ubc.ca</a>><br>
<b>Sent:</b> Wednesday, September 10, 2025 5:59 PM<br>
<b>To:</b> IOSIF PANAGIOTIS <<a href="mailto:PANAGIOTIS.IOSIF@units.it" target="_blank">PANAGIOTIS.IOSIF@units.it</a>><br>
<b>Cc:</b> Einstein Toolkit Users <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]</div>
<div style="direction:ltr"> </div>
</div>
<div style="direction:ltr;font-size:11pt">Hello Panayotis,<br>
<br>
"procs" as used by simfactory is (for historical reasons) a misleading<br>
term<br>
(<a id="m_-1333107120662898781OWA1b017ae2-8786-3124-ef9e-b8a7282873a3" href="https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html" target="_blank">https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html</a>).<br>
<br>
Really "procs" is the total number of threads that will be started,<br>
which usually will be the same as total number of (logical) cpus being<br>
used, and again usually the same as the number of (phyisal) cores used.<br>
<br>
If using --cores 16 you are asking for a total of 16 threads to be<br>
created which should be assigned to 16 cores. You should then select a<br>
number of threads-per-rank using the `--num-threads` option that is<br>
good for your setup. For 16 cores most likely `--num-threads 1` (ie one<br>
thread per MPI rank) is best. This will most likely fix your issue.<br>
<br>
Note that this will leave some cores unused and a number of simfactory<br>
machine description files will not handle that case gracefully since<br>
they assume that you will always use at least one full node.<br>
<br>
For you questions:<br>
<br>
* this is usually more of an issue with the SubmitScript and RunScript<br>
rather than the ini file and the simfactory command line, though<br>
in your case the init script has `num-threads = 56` which sets the<br>
default number of for `--num-threads` to 56, which is a bit high (but<br>
is overruled by a --num-threads option on your command line)<br>
* yes, specifying --num-threads on the command line will override the<br>
value in the ini file<br>
<br>
Yours,<br>
Roland<br>
<br>
<br>
> [CAUTION: Non-UBC Email]<br>
><br>
> Hi all,<br>
><br>
> After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.<br>
><br>
> However, I am getting the following warnings:<br>
> Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112)<br>
> Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads)<br>
> Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used)<br>
> From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.<br>
><br>
> Here is the command I use to run the TOV example:<br>
> ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00<br>
><br>
> And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's file<<a id="m_-1333107120662898781OWA67519326-77ab-3bc3-3869-eeb2c8afae43" href="https://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98" target="_blank">https://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98</a>>:<br>
><br>
> <a id="m_-1333107120662898781OWAa7dedd24-14d8-0f0c-dc80-520419d50f98" href="https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonardo-dcgp1.ini" target="_blank">
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonardo-dcgp1.ini</a><br>
><br>
> The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.<br>
><br>
> From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56.<br>
> Is that so?<br>
><br>
> So, my question is the following:<br>
><br>
> *<br>
> I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?<br>
> *<br>
> Or is there some option to specify when submitting the job that will override the cluster specs?<br>
><br>
> I read the simfactory's terminology explanation on nodes, threads etc<<a id="m_-1333107120662898781OWA412e04a4-fb0d-2cec-387c-dbd2db0b86c9" href="https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html" target="_blank">https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html</a>>,
but it is not clear to me how to request the cores I need.<br>
><br>
> Best,<br>
> Panayotis<br>
<br>
<br>
--<br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a id="m_-1333107120662898781OWA6412af53-9cea-7be2-20fa-661888f5530b" href="http://pgp.mit.edu" target="_blank">
http://pgp.mit.edu</a> .</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</div></blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p><font color="#000000">Pr<font face="arial, sans-serif">of. Bruno Giacomazzo<br>Deputy Director of the Department of Physics<br></font></font><span style="font-family:arial,sans-serif;color:rgb(0,0,0)">University of Milano-Bicocca<br></span><span style="font-family:arial,sans-serif;color:rgb(0,0,0)">Piazza della Scienza 3<br></span><span style="color:rgb(0,0,0)"><font face="arial, sans-serif">20126 Milano<br></font></span><span style="color:rgb(0,0,0);font-size:12.8px">Italy</span></p><p><font color="#000000"><span style="font-size:12.8px">email: </span><span style="font-size:12.8px"><a href="mailto:bruno.giacomazzo@unimib.it" target="_blank">bruno.giacomazzo@unimib.it</a><br></span><span style="font-size:12.8px">phone: (+39) 02 6448 2321</span></font><br><font color="#000000"><span style="font-size:12.8px">web: </span></font><span style="color:rgb(0,0,0);font-size:12.8px"><a href="http://www.brunogiacomazzo.org/" style="font-size:12.8px" target="_blank">http://www.brunogiacomazzo.org</a></span></p><p><font color="#000000">---------------------------------------------------------------------<br><span style="font-size:12.8px">There are only 10 types of people in the world:<br></span><span style="font-size:12.8px">Those who understand binary, and those who don't<br></span><span style="font-size:12.8px">----------------------------------------------------------------------<br><br></span></font></p></div></div></div></div></div>