<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 10/24/2025 9:16 AM, IOSIF PANAGIOTIS
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:GV1P250MB085762FC2CC965EC92BB7B92F9F1A@GV1P250MB0857.EURP250.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
Dear Steve,<br>
<br>
Thank you for your reply.<br>
<br>
At this point, I reckon it makes sense to just use a full node
(of 112 cores) for plain tests.<br>
In any case, they will finish quickly enough and not consume a
lot of my allocated time.<br>
</div>
</blockquote>
Probably so. Maybe even a fraction of a node.
<blockquote type="cite"
cite="mid:GV1P250MB085762FC2CC965EC92BB7B92F9F1A@GV1P250MB0857.EURP250.PROD.OUTLOOK.COM">
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof"><br>
I will leave the performance/convergence study varying the
number of cores for the time being. <br>
<br>
Regarding the --memory" option, I read the <a
title="https://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html"
href="https://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html"
moz-do-not-send="true">
minutes from the last ETK meeting</a>, and it seems that there
is no obvious answer.<br>
</div>
</blockquote>
<p>Yes. No one seems to know. I suspect there's some way to pass it
along to the submitscript where it can be given to slurm, but I'd
have to try and read through the source code to figure that out.</p>
<p><br>
</p>
<p>--Steve</p>
<blockquote type="cite"
cite="mid:GV1P250MB085762FC2CC965EC92BB7B92F9F1A@GV1P250MB0857.EURP250.PROD.OUTLOOK.COM">
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof"><br>
</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
In case it is useful, I summarise my current understanding
below:</div>
<ul
data-editing-info="{"applyListStyleFromLevel":false,"unorderedStyleType":2}"
style="margin-top: 0px; margin-bottom: 0px;">
<li
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof"><a
title="https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine"
href="https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine"
moz-do-not-send="true">this wiki link</a> says that "<code>memory</code> is
currently only used by simfactory's <code>distribute</code>
utility script"</div>
</li>
<li
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">The <code>distribute</code> script
seems to be a testing script only, so it does not seem to be
relevant when we submit a job</div>
</li>
<li
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">There doesn't
seem to be any source documenting exactly how simfactory
handles memory</div>
</li>
<li
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">Without taking
a look into the respective source code, my guess is that
simfactory will try to use as much memory as needed by the
job (but not necessarily the whole memory of the node)</div>
</li>
<li
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">In any case,
if we request a full node, we will get billed accordingly</div>
</li>
<li
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof"><i>Only if we
request less than a full node</i>, would the --memory
option possibly come into play and affect the billing</div>
</li>
</ul>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
<br>
</div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
Best,<br>
Panagiotis </div>
<div
style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"
class="elementToProof">
<br>
<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif"
style="font-size:11pt" color="#000000"><b>From:</b> Users
<a class="moz-txt-link-rfc2396E" href="mailto:users-bounces@einsteintoolkit.org"><users-bounces@einsteintoolkit.org></a> on behalf of Steven
Brandt via Users <a class="moz-txt-link-rfc2396E" href="mailto:users@einsteintoolkit.org"><users@einsteintoolkit.org></a><br>
<b>Sent:</b> Thursday, October 23, 2025 4:28 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:users@einsteintoolkit.org">users@einsteintoolkit.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:users@einsteintoolkit.org"><users@einsteintoolkit.org></a><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings:
cores/threads mismatch [Leonardo cluster]</font>
<div> </div>
</div>
<div>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 10/14/2025 3:55 AM, IOSIF
PANAGIOTIS wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style>
<div class="x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi all,<br>
<br>
I am sending a reminder regarding two unanswered questions
on the mailing list, in case someone has a suggestion.<br>
<br>
</div>
<ol
data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":3}"
start="1" style="margin-top:0px; margin-bottom:0px">
<li
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"1) "">
<div role="presentation" class="x_elementToProof">Clarification
about how SimFactory handles the "--memory" option and
how this affects how one should navigate the cluster's
billing policy:</div>
<div
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"1) "">
<div role="presentation" class="x_elementToProof"><a
id="LPlnk557853"
href="https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html"
class="x_moz-txt-link-freetext moz-txt-link-freetext"
moz-do-not-send="true">https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html</a></div>
</div>
<div
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"1) "">
<div role="presentation" class="x_elementToProof"><br>
</div>
</div>
</li>
<li
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
<div role="presentation" class="x_elementToProof">Using
'leonardo-dcgp.ini' and understanding how to properly
request one full node:</div>
<div
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
<div role="presentation" class="x_elementToProof"><a
id="LPlnk416851"
href="https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html"
class="x_moz-txt-link-freetext moz-txt-link-freetext"
moz-do-not-send="true">https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html</a></div>
</div>
</li>
</ol>
</blockquote>
<p>Normally, one requests --procs equal to the number of cores
on the node.</p>
<p><br>
</p>
<p>So, imagine one has a machine with nodes that have 32 cores
each.</p>
<p><br>
</p>
<p>One could say --procs 32, and that should be an entire node.
However, maybe you want to run with 8 threads per MPI task. In
that case, you would say --procs 32 --num-threads 8.</p>
<p><br>
</p>
<p>If you want to run on N nodes, then the number of procs would
be 32*N, and Simfactory will figure it out.</p>
<p><br>
</p>
<p>--Steve</p>
<blockquote type="cite">
<ol
data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":3}"
start="1" style="margin-top:0px; margin-bottom:0px">
<li
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
<div
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
</div>
<br>
</li>
</ol>
<div class="x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks,</div>
<div class="x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Panayotis</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
face="Calibri, sans-serif" color="#000000"
style="font-size:11pt"><b>From:</b> Users
<a class="x_moz-txt-link-rfc2396E"
href="mailto:users-bounces@einsteintoolkit.org"
moz-do-not-send="true">
<users-bounces@einsteintoolkit.org></a> on behalf
of IOSIF PANAGIOTIS <a class="x_moz-txt-link-rfc2396E"
href="mailto:PANAGIOTIS.IOSIF@units.it"
moz-do-not-send="true">
<PANAGIOTIS.IOSIF@units.it></a><br>
<b>Sent:</b> Monday, September 29, 2025 12:24 PM<br>
<b>To:</b> Roland Haas <a class="x_moz-txt-link-rfc2396E"
href="mailto:rhaas@mail.ubc.ca" moz-do-not-send="true">
<rhaas@mail.ubc.ca></a>; Bruno Giacomazzo <a
class="x_moz-txt-link-rfc2396E"
href="mailto:bruno.giacomazzo@unimib.it"
moz-do-not-send="true">
<bruno.giacomazzo@unimib.it></a><br>
<b>Cc:</b> Einstein Toolkit Users <a
class="x_moz-txt-link-rfc2396E"
href="mailto:users@einsteintoolkit.org"
moz-do-not-send="true">
<users@einsteintoolkit.org></a><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings:
cores/threads mismatch [Leonardo cluster]</font>
<div> </div>
</div>
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style>
<div dir="ltr">
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Roland,<br>
<br>
Thanks for your reply.<br>
You touch on an important point, i.e the <b>cluster's</b>
<b>billing policy</b>, that hadn't crossed my mind.<br>
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
From the billing policy of Leonardo, it seems that <b>it
is</b> <b>possible to use only a fraction of a node's
total CPUs.</b></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<a
class="x_x_OWAAutoLink x_moz-txt-link-freetext moz-txt-link-freetext"
id="LPlnk971574"
href="https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy"
moz-do-not-send="true">https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy</a></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>However</b>, the documentation also stresses that:</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<i>...if a job reserves all of a node’s RAM — even without
utilizing all its CPUs — the node becomes unusable for
other jobs and is therefore billed accordingly.<br>
<br>
</i></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
So, apart from the cores requested, <b>should I also try
to calculate the RAM requirements?</b></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
For example, I see that Bruno's "leonardo-dcgp.ini" file
specifies:</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<code>memory = 494000</code></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
And the respective submitscript also has this line:</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<code>#SBATCH --mem 494000MB</code></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I note that each node in Leonardo has 512GB of RAM, so
that means that <b>the script requests ~94.2% of the RAM.</b></div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am not sure I follow the reasoning behind this.<br>
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
What is the default behavior of SimFactory if I were to
remove the above specifications from the config files?</div>
<div class="x_x_elementToProof"
style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Because, if by default Simfactory requests/uses all the
RAM available in a node, then as far as I understand, it
does not make sense to request fewer cores than a full
node.<br>
Let me know what you think.<br>
<br>
Best,<br>
Panayotis <br>
<br>
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font
face="Calibri, sans-serif" color="#000000"
style="font-size:11pt"><b>From:</b> Roland Haas
<a class="x_moz-txt-link-rfc2396E"
href="mailto:rhaas@mail.ubc.ca" moz-do-not-send="true"><rhaas@mail.ubc.ca></a><br>
<b>Sent:</b> Friday, September 26, 2025 4:31 PM<br>
<b>To:</b> Bruno Giacomazzo <a
class="x_moz-txt-link-rfc2396E"
href="mailto:bruno.giacomazzo@unimib.it"
moz-do-not-send="true">
<bruno.giacomazzo@unimib.it></a><br>
<b>Cc:</b> IOSIF PANAGIOTIS <a
class="x_moz-txt-link-rfc2396E"
href="mailto:PANAGIOTIS.IOSIF@units.it"
moz-do-not-send="true">
<PANAGIOTIS.IOSIF@units.it></a>; Einstein
Toolkit Users <a class="x_moz-txt-link-rfc2396E"
href="mailto:users@einsteintoolkit.org"
moz-do-not-send="true">
<users@einsteintoolkit.org></a><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings:
cores/threads mismatch [Leonardo cluster]</font>
<div> </div>
</div>
<div class="x_x_BodyFragment"><font size="2"><span
style="font-size:11pt">
<div class="x_x_PlainText">Hello all,<br>
<br>
> I never used --cores and I don't know the
difference with procs.<br>
<br>
--cores is a synonym for --procs in simfactory. The
hope was to avoid<br>
the confusion of "procs" being "Processes" or
"Processors". Though it<br>
has been pointed out that the best name would
actually be "--threads"<br>
since that is what simfactory actually starts, which
then collides with<br>
"--num-threads" (threads per process).<br>
<br>
Does Leonardo actually charge you for partial nodes
if you do no use a<br>
full one? Simfactory is mostly written under the
assumption (true at<br>
the time) that HPC systems would give you full nodes
all the time, so<br>
if you use 1 core or 112 cores of a node, the charge
would be the same<br>
(though shared node systems are becoming more common
for HPC now [or<br>
again]).<br>
<br>
Yours,<br>
Roland<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore
support encrypting<br>
and signing email messages. Get my PGP key from <a
href="http://pgp.mit.edu"
class="x_moz-txt-link-freetext moz-txt-link-freetext"
moz-do-not-send="true">
http://pgp.mit.edu</a> .<br>
</div>
</span></font></div>
</div>
<br>
<fieldset class="x_moz-mime-attachment-header"></fieldset>
<pre class="x_moz-quote-pre">_______________________________________________
Users mailing list
<a class="x_moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:Users@einsteintoolkit.org" moz-do-not-send="true">Users@einsteintoolkit.org</a>
<a class="x_moz-txt-link-freetext moz-txt-link-freetext"
href="http://lists.einsteintoolkit.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre wrap="" class="moz-quote-pre">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a>
<a class="moz-txt-link-freetext" href="http://lists.einsteintoolkit.org/mailman/listinfo/users">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</body>
</html>