<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Dear Steve,<br>
<br>
Thank you for your reply.<br>
<br>
At this point, I reckon it makes sense to just use a full node (of 112 cores) for plain tests.<br>
In any case, they will finish quickly enough and not consume a lot of my allocated time.<br>
<br>
I will leave the performance/convergence study varying the number of cores for the time being. <br>
<br>
Regarding the --memory" option, I read the <a title="https://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html" href="https://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html">
minutes from the last ETK meeting</a>, and it seems that there is no obvious answer.<br>
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
In case it is useful, I summarise my current understanding below:</div>
<ul data-editing-info="{"applyListStyleFromLevel":false,"unorderedStyleType":2}" style="margin-top: 0px; margin-bottom: 0px;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof"><a title="https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine" href="https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine">this wiki link</a> says that "<code>memory</code> is currently
only used by simfactory's <code>distribute</code> utility script"</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">The <code>distribute</code> script seems to be a testing script only, so it does not seem to be relevant when we submit a job</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">There doesn't seem to be any source documenting exactly how simfactory handles memory</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">Without taking a look into the respective source code, my guess is that simfactory will try to use as much memory as needed by the job (but not necessarily the whole memory of the node)</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof">In any case, if we request a full node, we will get billed accordingly</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div role="presentation" class="elementToProof"><i>Only if we request less than a full node</i>, would the --memory option possibly come into play and affect the billing</div>
</li></ul>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Best,<br>
Panagiotis </div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Users <users-bounces@einsteintoolkit.org> on behalf of Steven Brandt via Users <users@einsteintoolkit.org><br>
<b>Sent:</b> Thursday, October 23, 2025 4:28 PM<br>
<b>To:</b> users@einsteintoolkit.org <users@einsteintoolkit.org><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]</font>
<div> </div>
</div>
<div>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote:<br>
</div>
<blockquote type="cite"><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi all,<br>
<br>
I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion.<br>
<br>
</div>
<ol data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":3}" start="1" style="margin-top:0px; margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"1) "">
<div role="presentation" class="x_elementToProof">Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"1) "">
<div role="presentation" class="x_elementToProof"><a id="LPlnk557853" href="https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html" class="x_moz-txt-link-freetext">https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html</a></div>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"1) "">
<div role="presentation" class="x_elementToProof"><br>
</div>
</div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
<div role="presentation" class="x_elementToProof">Using 'leonardo-dcgp.ini' and understanding how to properly request one full node:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
<div role="presentation" class="x_elementToProof"><a id="LPlnk416851" href="https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html" class="x_moz-txt-link-freetext">https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html</a></div>
</div>
</li></ol>
</blockquote>
<p>Normally, one requests --procs equal to the number of cores on the node.</p>
<p><br>
</p>
<p>So, imagine one has a machine with nodes that have 32 cores each.</p>
<p><br>
</p>
<p>One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8.</p>
<p><br>
</p>
<p>If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out.</p>
<p><br>
</p>
<p>--Steve</p>
<blockquote type="cite">
<ol data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":3}" start="1" style="margin-top:0px; margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); list-style-type:"2) "">
</div>
<br>
</li></ol>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks,</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Panayotis</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Users
<a class="x_moz-txt-link-rfc2396E" href="mailto:users-bounces@einsteintoolkit.org">
<users-bounces@einsteintoolkit.org></a> on behalf of IOSIF PANAGIOTIS <a class="x_moz-txt-link-rfc2396E" href="mailto:PANAGIOTIS.IOSIF@units.it">
<PANAGIOTIS.IOSIF@units.it></a><br>
<b>Sent:</b> Monday, September 29, 2025 12:24 PM<br>
<b>To:</b> Roland Haas <a class="x_moz-txt-link-rfc2396E" href="mailto:rhaas@mail.ubc.ca">
<rhaas@mail.ubc.ca></a>; Bruno Giacomazzo <a class="x_moz-txt-link-rfc2396E" href="mailto:bruno.giacomazzo@unimib.it">
<bruno.giacomazzo@unimib.it></a><br>
<b>Cc:</b> Einstein Toolkit Users <a class="x_moz-txt-link-rfc2396E" href="mailto:users@einsteintoolkit.org">
<users@einsteintoolkit.org></a><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Roland,<br>
<br>
Thanks for your reply.<br>
You touch on an important point, i.e the <b>cluster's</b> <b>billing policy</b>, that hadn't crossed my mind.<br>
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
>From the billing policy of Leonardo, it seems that <b>it is</b> <b>possible to use only a fraction of a node's total CPUs.</b></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<a class="x_x_OWAAutoLink x_moz-txt-link-freetext" id="LPlnk971574" href="https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy">https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy</a></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>However</b>, the documentation also stresses that:</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<i>...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.<br>
<br>
</i></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
So, apart from the cores requested, <b>should I also try to calculate the RAM requirements?</b></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
For example, I see that Bruno's "leonardo-dcgp.ini" file specifies:</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<code>memory = 494000</code></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
And the respective submitscript also has this line:</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<code>#SBATCH --mem 494000MB</code></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I note that each node in Leonardo has 512GB of RAM, so that means that <b>the script requests ~94.2% of the RAM.</b></div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am not sure I follow the reasoning behind this.<br>
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
What is the default behavior of SimFactory if I were to remove the above specifications from the config files?</div>
<div class="x_x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node.<br>
Let me know what you think.<br>
<br>
Best,<br>
Panayotis <br>
<br>
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Roland Haas
<a class="x_moz-txt-link-rfc2396E" href="mailto:rhaas@mail.ubc.ca"><rhaas@mail.ubc.ca></a><br>
<b>Sent:</b> Friday, September 26, 2025 4:31 PM<br>
<b>To:</b> Bruno Giacomazzo <a class="x_moz-txt-link-rfc2396E" href="mailto:bruno.giacomazzo@unimib.it">
<bruno.giacomazzo@unimib.it></a><br>
<b>Cc:</b> IOSIF PANAGIOTIS <a class="x_moz-txt-link-rfc2396E" href="mailto:PANAGIOTIS.IOSIF@units.it">
<PANAGIOTIS.IOSIF@units.it></a>; Einstein Toolkit Users <a class="x_moz-txt-link-rfc2396E" href="mailto:users@einsteintoolkit.org">
<users@einsteintoolkit.org></a><br>
<b>Subject:</b> Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]</font>
<div> </div>
</div>
<div class="x_x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_x_PlainText">Hello all,<br>
<br>
> I never used --cores and I don't know the difference with procs.<br>
<br>
--cores is a synonym for --procs in simfactory. The hope was to avoid<br>
the confusion of "procs" being "Processes" or "Processors". Though it<br>
has been pointed out that the best name would actually be "--threads"<br>
since that is what simfactory actually starts, which then collides with<br>
"--num-threads" (threads per process).<br>
<br>
Does Leonardo actually charge you for partial nodes if you do no use a<br>
full one? Simfactory is mostly written under the assumption (true at<br>
the time) that HPC systems would give you full nodes all the time, so<br>
if you use 1 core or 112 cores of a node, the charge would be the same<br>
(though shared node systems are becoming more common for HPC now [or<br>
again]).<br>
<br>
Yours,<br>
Roland<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" class="x_moz-txt-link-freetext">
http://pgp.mit.edu</a> .<br>
</div>
</span></font></div>
</div>
<br>
<fieldset class="x_moz-mime-attachment-header"></fieldset>
<pre class="x_moz-quote-pre">_______________________________________________
Users mailing list
<a class="x_moz-txt-link-abbreviated" href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a>
<a class="x_moz-txt-link-freetext" href="http://lists.einsteintoolkit.org/mailman/listinfo/users">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
</body>
</html>