[Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

Fri Oct 24 09:16:48 CDT 2025

Dear Steve,

Thank you for your reply.

At this point, I reckon it makes sense to just use a full node (of 112 cores) for plain tests.
In any case, they will finish quickly enough and not consume a lot of my allocated time.

I will leave the performance/convergence study varying the number of cores for the time being.

Regarding the --memory" option, I read the minutes from the last ETK meeting<https://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html>, and it seems that there is no obvious answer.

In case it is useful, I summarise my current understanding below:

  *
this wiki link<https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine> says that "memory is currently only used by simfactory's distribute utility script"
  *
The distribute script seems to be a testing script only, so it does not seem to be relevant when we submit a job
  *
There doesn't seem to be any source documenting exactly how simfactory handles memory
  *
Without taking a look into the respective source code, my guess is that simfactory will try to use as much memory as needed by the job (but not necessarily the whole memory of the node)
  *
In any case, if we request a full node, we will  get billed accordingly
  *
Only if we request less than a full node, would the --memory option possibly come into play and affect the billing

Best,
Panagiotis

________________________________
From: Users <users-bounces at einsteintoolkit.org> on behalf of Steven Brandt via Users <users at einsteintoolkit.org>
Sent: Thursday, October 23, 2025 4:28 PM
To: users at einsteintoolkit.org <users at einsteintoolkit.org>
Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote:
Hi all,

I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion.

  1.
Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy:
https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html

  2.
Using 'leonardo-dcgp.ini' and understanding how to properly request one full node:
https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html

Normally, one requests --procs equal to the number of cores on the node.

So, imagine one has a machine with nodes that have 32 cores each.

One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8.

If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out.

--Steve

  1.

Thanks,
Panayotis
________________________________
From: Users <users-bounces at einsteintoolkit.org><mailto:users-bounces at einsteintoolkit.org> on behalf of IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF at units.it><mailto:PANAGIOTIS.IOSIF at units.it>
Sent: Monday, September 29, 2025 12:24 PM
To: Roland Haas <rhaas at mail.ubc.ca><mailto:rhaas at mail.ubc.ca>; Bruno Giacomazzo <bruno.giacomazzo at unimib.it><mailto:bruno.giacomazzo at unimib.it>
Cc: Einstein Toolkit Users <users at einsteintoolkit.org><mailto:users at einsteintoolkit.org>
Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

Hi Roland,

Thanks for your reply.
You touch on an important point, i.e the cluster's billing policy, that hadn't crossed my mind.

>From the billing policy of Leonardo, it seems that it is possible to use only a fraction of a node's total CPUs.
https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy

However, the documentation also stresses that:
...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.

So, apart from the cores requested, should I also try to calculate the RAM requirements?
For example, I see that Bruno's "leonardo-dcgp.ini" file specifies:
memory = 494000
And the respective submitscript also has this line:
#SBATCH --mem 494000MB

I note that each node in Leonardo has 512GB of RAM, so that means that the script requests ~94.2% of the RAM.
I am not sure I follow the reasoning behind this.

What is the default behavior of SimFactory if I were to remove the above specifications from the config files?
Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node.
Let me know what you think.

Best,
Panayotis

________________________________
From: Roland Haas <rhaas at mail.ubc.ca><mailto:rhaas at mail.ubc.ca>
Sent: Friday, September 26, 2025 4:31 PM
To: Bruno Giacomazzo <bruno.giacomazzo at unimib.it><mailto:bruno.giacomazzo at unimib.it>
Cc: IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF at units.it><mailto:PANAGIOTIS.IOSIF at units.it>; Einstein Toolkit Users <users at einsteintoolkit.org><mailto:users at einsteintoolkit.org>
Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]

Hello all,

> I never used --cores and I don't know the difference with procs.

--cores is a synonym for --procs in simfactory. The hope was to avoid
the confusion of "procs" being "Processes" or "Processors". Though it
has been pointed out that the best name would actually be "--threads"
since that is what simfactory actually starts, which then collides with
"--num-threads" (threads per process).

Does Leonardo actually charge you for partial nodes if you do no use a
full one? Simfactory is mostly written under the assumption (true at
the time) that HPC systems would give you full nodes all the time, so
if you use 1 core or 112 cores of a node, the charge would be the same
(though shared node systems are becoming more common for HPC now [or
again]).

Yours,
Roland

--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .

_______________________________________________
Users mailing list
Users at einsteintoolkit.org<mailto:Users at einsteintoolkit.org>
http://lists.einsteintoolkit.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20251024/e228d29d/attachment-0001.htm>