Hi all,
After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.
However, I am getting the following warnings: Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112) Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used)
From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.
Here is the command I use to run the TOV example: ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00
And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's filehttps://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98:
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonard...
The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.
From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56.
Is that so?
So, my question is the following:
* I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file? * Or is there some option to specify when submitting the job that will override the cluster specs?
I read the simfactory's terminology explanation on nodes, threads etchttps://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html, but it is not clear to me how to request the cores I need.
Best, Panayotis
Hello Panayotis,
"procs" as used by simfactory is (for historical reasons) a misleading term (https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.htm...).
Really "procs" is the total number of threads that will be started, which usually will be the same as total number of (logical) cpus being used, and again usually the same as the number of (phyisal) cores used.
If using --cores 16 you are asking for a total of 16 threads to be created which should be assigned to 16 cores. You should then select a number of threads-per-rank using the `--num-threads` option that is good for your setup. For 16 cores most likely `--num-threads 1` (ie one thread per MPI rank) is best. This will most likely fix your issue.
Note that this will leave some cores unused and a number of simfactory machine description files will not handle that case gracefully since they assume that you will always use at least one full node.
For you questions:
* this is usually more of an issue with the SubmitScript and RunScript rather than the ini file and the simfactory command line, though in your case the init script has `num-threads = 56` which sets the default number of for `--num-threads` to 56, which is a bit high (but is overruled by a --num-threads option on your command line) * yes, specifying --num-threads on the command line will override the value in the ini file
Yours, Roland
[CAUTION: Non-UBC Email]
Hi all,
After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.
However, I am getting the following warnings: Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112) Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used) From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.
Here is the command I use to run the TOV example: ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00
And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's filehttps://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98:
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonard...
The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.
From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56. Is that so?
So, my question is the following:
I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?
Or is there some option to specify when submitting the job that will override the cluster specs?
I read the simfactory's terminology explanation on nodes, threads etchttps://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html, but it is not clear to me how to request the cores I need.
Best, Panayotis
Dear Roland,
I am following up on my question about how to correctly request the resources that I need.
Here is a reminder of the context:
* I have installed ETK on the Leonardo, DCGP partition at CINECA * I want to run a TOV star example requesting a different number of cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.
*
I use this command to submit the simulation (sim is the typical alias for "./simfactory/bin/sim"):
* sim create-submit tov_test --parfile par/tov_ET.par --cores 16 --num-threads 1 --walltime 00:20:00
The config files I use are here, and they are based on Bruno Giacomazzo's originals:
* https://github.com/piosif/einstein-toolkit-config-leonardo/
I get a weird error from the CarpetLib thorn, which complains that:
* The grid structure is inconsistent. It is impossible to continue.
I have attached the simulation .err and .out files.
FWIW, inspecting the very beginning of the two files, it seems that the overall setup I used (config files and submit command) results in:
* MPI processes: 16 * OpenMP threads per core: 0.14285714285714285
This is 16 divided by 112 (the total cores in each node), and it does not seem to be what I want. I would expect that each OpenMP thread should be assigned to one core. Furthermore, for that simple test and if my understanding of the basics of MPI and OpenMP is correct, it would make more sense to have just one MPI process and 16 OpenMP threads.
Could you help me understand what is wrong with my submit command above in connection to the options specified in my config files?
Best, Panayotis
________________________________ From: Roland Haas rhaas@mail.ubc.ca Sent: Wednesday, September 10, 2025 5:59 PM To: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello Panayotis,
"procs" as used by simfactory is (for historical reasons) a misleading term (https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.htm...).
Really "procs" is the total number of threads that will be started, which usually will be the same as total number of (logical) cpus being used, and again usually the same as the number of (phyisal) cores used.
If using --cores 16 you are asking for a total of 16 threads to be created which should be assigned to 16 cores. You should then select a number of threads-per-rank using the `--num-threads` option that is good for your setup. For 16 cores most likely `--num-threads 1` (ie one thread per MPI rank) is best. This will most likely fix your issue.
Note that this will leave some cores unused and a number of simfactory machine description files will not handle that case gracefully since they assume that you will always use at least one full node.
For you questions:
* this is usually more of an issue with the SubmitScript and RunScript rather than the ini file and the simfactory command line, though in your case the init script has `num-threads = 56` which sets the default number of for `--num-threads` to 56, which is a bit high (but is overruled by a --num-threads option on your command line) * yes, specifying --num-threads on the command line will override the value in the ini file
Yours, Roland
[CAUTION: Non-UBC Email]
Hi all,
After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.
However, I am getting the following warnings: Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112) Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used) From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.
Here is the command I use to run the TOV example: ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00
And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's filehttps://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98:
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonard...
The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.
From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56. Is that so?
So, my question is the following:
I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?
Or is there some option to specify when submitting the job that will override the cluster specs?
I read the simfactory's terminology explanation on nodes, threads etchttps://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html, but it is not clear to me how to request the cores I need.
Best, Panayotis
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hi all,
I am bumping this follow-up question regarding inconsistency warnings when requesting resources during a job submit, in case someone has some suggestion.
Thanks, Panayotis
________________________________ From: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it Sent: Thursday, September 18, 2025 3:25 PM To: Roland Haas rhaas@mail.ubc.ca Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Dear Roland,
I am following up on my question about how to correctly request the resources that I need.
Here is a reminder of the context:
* I have installed ETK on the Leonardo, DCGP partition at CINECA * I want to run a TOV star example requesting a different number of cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.
*
I use this command to submit the simulation (sim is the typical alias for "./simfactory/bin/sim"):
* sim create-submit tov_test --parfile par/tov_ET.par --cores 16 --num-threads 1 --walltime 00:20:00
The config files I use are here, and they are based on Bruno Giacomazzo's originals:
* https://github.com/piosif/einstein-toolkit-config-leonardo/
I get a weird error from the CarpetLib thorn, which complains that:
* The grid structure is inconsistent. It is impossible to continue.
I have attached the simulation .err and .out files.
FWIW, inspecting the very beginning of the two files, it seems that the overall setup I used (config files and submit command) results in:
* MPI processes: 16 * OpenMP threads per core: 0.14285714285714285
This is 16 divided by 112 (the total cores in each node), and it does not seem to be what I want. I would expect that each OpenMP thread should be assigned to one core. Furthermore, for that simple test and if my understanding of the basics of MPI and OpenMP is correct, it would make more sense to have just one MPI process and 16 OpenMP threads.
Could you help me understand what is wrong with my submit command above in connection to the options specified in my config files?
Best, Panayotis
________________________________ From: Roland Haas rhaas@mail.ubc.ca Sent: Wednesday, September 10, 2025 5:59 PM To: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello Panayotis,
"procs" as used by simfactory is (for historical reasons) a misleading term (https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.htm...).
Really "procs" is the total number of threads that will be started, which usually will be the same as total number of (logical) cpus being used, and again usually the same as the number of (phyisal) cores used.
If using --cores 16 you are asking for a total of 16 threads to be created which should be assigned to 16 cores. You should then select a number of threads-per-rank using the `--num-threads` option that is good for your setup. For 16 cores most likely `--num-threads 1` (ie one thread per MPI rank) is best. This will most likely fix your issue.
Note that this will leave some cores unused and a number of simfactory machine description files will not handle that case gracefully since they assume that you will always use at least one full node.
For you questions:
* this is usually more of an issue with the SubmitScript and RunScript rather than the ini file and the simfactory command line, though in your case the init script has `num-threads = 56` which sets the default number of for `--num-threads` to 56, which is a bit high (but is overruled by a --num-threads option on your command line) * yes, specifying --num-threads on the command line will override the value in the ini file
Yours, Roland
[CAUTION: Non-UBC Email]
Hi all,
After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.
However, I am getting the following warnings: Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112) Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used) From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.
Here is the command I use to run the TOV example: ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00
And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's filehttps://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98:
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonard...
The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.
From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56. Is that so?
So, my question is the following:
I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?
Or is there some option to specify when submitting the job that will override the cluster specs?
I read the simfactory's terminology explanation on nodes, threads etchttps://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html, but it is not clear to me how to request the cores I need.
Best, Panayotis
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Panayotis, Did you try using --procs=16 --num-threads=1? I never used --cores and I don't know the difference with procs.
Cheers, Bruno
Il giorno gio 25 set 2025 alle ore 17:29 IOSIF PANAGIOTIS < PANAGIOTIS.IOSIF@units.it> ha scritto:
Hi all,
I am bumping this follow-up question regarding inconsistency warnings when requesting resources during a job submit, in case someone has some suggestion.
Thanks, Panayotis
*From:* IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it *Sent:* Thursday, September 18, 2025 3:25 PM *To:* Roland Haas rhaas@mail.ubc.ca *Cc:* Einstein Toolkit Users users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Dear Roland,
I am following up on my question about how to correctly request the resources that I need.
Here is a reminder of the context:
- I have installed ETK on the Leonardo, DCGP partition at CINECA
- I want to run a TOV star example requesting a different number of
cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.
I use this command to submit the simulation (sim is the typical alias for "./simfactory/bin/sim"):
- sim create-submit tov_test --parfile par/tov_ET.par --cores 16
--num-threads 1 --walltime 00:20:00
The config files I use are here, and they are based on Bruno Giacomazzo's originals:
I get a *weird error from the CarpetLib thorn*, which complains that:
- *The grid structure is inconsistent. It is impossible to continue.*
I have attached the simulation .err and .out files.
FWIW, inspecting the very beginning of the two files, it seems that the overall setup I used (config files and submit command) results in:
- MPI processes: 16
- OpenMP threads per core: 0.14285714285714285
This is 16 divided by 112 (the total cores in each node), and it does not seem to be what I want. I would expect that each OpenMP thread should be assigned to one core. Furthermore, for that simple test and if my understanding of the basics of MPI and OpenMP is correct, it would make more sense to have just one MPI process and 16 OpenMP threads.
Could you help me understand what is wrong with my submit command above in connection to the options specified in my config files?
Best, Panayotis
*From:* Roland Haas rhaas@mail.ubc.ca *Sent:* Wednesday, September 10, 2025 5:59 PM *To:* IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it *Cc:* Einstein Toolkit Users users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello Panayotis,
"procs" as used by simfactory is (for historical reasons) a misleading term ( https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.htm... ).
Really "procs" is the total number of threads that will be started, which usually will be the same as total number of (logical) cpus being used, and again usually the same as the number of (phyisal) cores used.
If using --cores 16 you are asking for a total of 16 threads to be created which should be assigned to 16 cores. You should then select a number of threads-per-rank using the `--num-threads` option that is good for your setup. For 16 cores most likely `--num-threads 1` (ie one thread per MPI rank) is best. This will most likely fix your issue.
Note that this will leave some cores unused and a number of simfactory machine description files will not handle that case gracefully since they assume that you will always use at least one full node.
For you questions:
- this is usually more of an issue with the SubmitScript and RunScript rather than the ini file and the simfactory command line, though in your case the init script has `num-threads = 56` which sets the default number of for `--num-threads` to 56, which is a bit high (but is overruled by a --num-threads option on your command line)
- yes, specifying --num-threads on the command line will override the value in the ini file
Yours, Roland
[CAUTION: Non-UBC Email]
Hi all,
After having installed the toolkit on Leonardo, CINECA, I wanted to try
the TOV examples.
However, I am getting the following warnings: Warning: Too many threads per process specified: specified
num-threads=56 (ppn-used is 112)
Warning: Total number of threads and number of threads per process are
inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads)
Warning: Total number of threads and number of cores per node are
inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used)
From those, it seems that I am doing something wrong either in my
machine.ini file or in the resources request when submitting the job.
Here is the command I use to run the TOV example: ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16
--walltime 00:20:00
And here is the ini file I am using (the "Cluster characteristics"
section is identical to Bruno Giacomazzo's file< https://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e0...
:
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonard...
The job ran and completed fine, but from what I understand, it ran on 56
cores instead of the 16 that I requested.
From what I understand, the problem is that I am requesting just 16
cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56.
Is that so?
So, my question is the following:
I want to do some tests, running a single star e.g. on a single node
using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?
Or is there some option to specify when submitting the job that will
override the cluster specs?
I read the simfactory's terminology explanation on nodes, threads etc<
https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.htm..., but it is not clear to me how to request the cores I need.
Best, Panayotis
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Bruno,
I tried with --procs instead of --cores, but I got the same error (see .err and .out files attached).
Checking the options available via: simfactory/bin/sim --help
it seems that these two options have absolutely no difference between them: --cores=<int> Total number of threads to use --procs=<int> See --cores
In principle, of course, I could work with all the 112 cores of one node on Leonardo.
However, this seems a waste of resources for a simple test. And in any case, it would be good to understand how to properly request resources using the different options of SimFactory, overriding the mdb entries if need be.
I will experiment further with the options, but if you have any other suggestion it is welcome.
Best, Panayotis
________________________________ From: Bruno Giacomazzo bruno.giacomazzo@unimib.it Sent: Friday, September 26, 2025 10:53 AM To: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Panayotis, Did you try using --procs=16 --num-threads=1? I never used --cores and I don't know the difference with procs.
Cheers, Bruno
Il giorno gio 25 set 2025 alle ore 17:29 IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF@units.itmailto:PANAGIOTIS.IOSIF@units.it> ha scritto: Hi all,
I am bumping this follow-up question regarding inconsistency warnings when requesting resources during a job submit, in case someone has some suggestion.
Thanks, Panayotis
________________________________ From: IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF@units.itmailto:PANAGIOTIS.IOSIF@units.it> Sent: Thursday, September 18, 2025 3:25 PM To: Roland Haas <rhaas@mail.ubc.camailto:rhaas@mail.ubc.ca> Cc: Einstein Toolkit Users <users@einsteintoolkit.orgmailto:users@einsteintoolkit.org> Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Dear Roland,
I am following up on my question about how to correctly request the resources that I need.
Here is a reminder of the context:
* I have installed ETK on the Leonardo, DCGP partition at CINECA * I want to run a TOV star example requesting a different number of cores each time (e.g. 16, 32, 64 etc) to better understand the timescales.
*
I use this command to submit the simulation (sim is the typical alias for "./simfactory/bin/sim"):
* sim create-submit tov_test --parfile par/tov_ET.par --cores 16 --num-threads 1 --walltime 00:20:00
The config files I use are here, and they are based on Bruno Giacomazzo's originals:
* https://github.com/piosif/einstein-toolkit-config-leonardo/
I get a weird error from the CarpetLib thorn, which complains that:
* The grid structure is inconsistent. It is impossible to continue.
I have attached the simulation .err and .out files.
FWIW, inspecting the very beginning of the two files, it seems that the overall setup I used (config files and submit command) results in:
* MPI processes: 16 * OpenMP threads per core: 0.14285714285714285
This is 16 divided by 112 (the total cores in each node), and it does not seem to be what I want. I would expect that each OpenMP thread should be assigned to one core. Furthermore, for that simple test and if my understanding of the basics of MPI and OpenMP is correct, it would make more sense to have just one MPI process and 16 OpenMP threads.
Could you help me understand what is wrong with my submit command above in connection to the options specified in my config files?
Best, Panayotis
________________________________ From: Roland Haas <rhaas@mail.ubc.camailto:rhaas@mail.ubc.ca> Sent: Wednesday, September 10, 2025 5:59 PM To: IOSIF PANAGIOTIS <PANAGIOTIS.IOSIF@units.itmailto:PANAGIOTIS.IOSIF@units.it> Cc: Einstein Toolkit Users <users@einsteintoolkit.orgmailto:users@einsteintoolkit.org> Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello Panayotis,
"procs" as used by simfactory is (for historical reasons) a misleading term (https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.htm...).
Really "procs" is the total number of threads that will be started, which usually will be the same as total number of (logical) cpus being used, and again usually the same as the number of (phyisal) cores used.
If using --cores 16 you are asking for a total of 16 threads to be created which should be assigned to 16 cores. You should then select a number of threads-per-rank using the `--num-threads` option that is good for your setup. For 16 cores most likely `--num-threads 1` (ie one thread per MPI rank) is best. This will most likely fix your issue.
Note that this will leave some cores unused and a number of simfactory machine description files will not handle that case gracefully since they assume that you will always use at least one full node.
For you questions:
* this is usually more of an issue with the SubmitScript and RunScript rather than the ini file and the simfactory command line, though in your case the init script has `num-threads = 56` which sets the default number of for `--num-threads` to 56, which is a bit high (but is overruled by a --num-threads option on your command line) * yes, specifying --num-threads on the command line will override the value in the ini file
Yours, Roland
[CAUTION: Non-UBC Email]
Hi all,
After having installed the toolkit on Leonardo, CINECA, I wanted to try the TOV examples.
However, I am getting the following warnings: Warning: Too many threads per process specified: specified num-threads=56 (ppn-used is 112) Warning: Total number of threads and number of threads per process are inconsistent: procs=16, num-threads=56 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=16, ppn-used=112 (procs must be an integer multiple of ppn-used) From those, it seems that I am doing something wrong either in my machine.ini file or in the resources request when submitting the job.
Here is the command I use to run the TOV example: ./simfactory/bin/sim submit tov_ET --parfile par/tov_ET.par --cores 16 --walltime 00:20:00
And here is the ini file I am using (the "Cluster characteristics" section is identical to Bruno Giacomazzo's filehttps://bitbucket.org/simfactory/simfactory2/commits/0d4011bf554ee695f9543e048ae9ff5edc5b5c98:
https://github.com/piosif/einstein-toolkit-config-leonardo/blob/main/leonard...
The job ran and completed fine, but from what I understand, it ran on 56 cores instead of the 16 that I requested.
From what I understand, the problem is that I am requesting just 16 cores for this simple test, but due to the cluster specs in the ini file, SimFactory expects me to request a multiple of 56. Is that so?
So, my question is the following:
I want to do some tests, running a single star e.g. on a single node using 16, 32, 64 cores (to understand the timescale required). Do I need to change my ini file?
Or is there some option to specify when submitting the job that will override the cluster specs?
I read the simfactory's terminology explanation on nodes, threads etchttps://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html, but it is not clear to me how to request the cores I need.
Best, Panayotis
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . _______________________________________________ Users mailing list Users@einsteintoolkit.orgmailto:Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Prof. Bruno Giacomazzo Deputy Director of the Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy
email: bruno.giacomazzo@unimib.itmailto:bruno.giacomazzo@unimib.it phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.orghttp://www.brunogiacomazzo.org/
--------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------
Hello all,
I never used --cores and I don't know the difference with procs.
--cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process).
Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]).
Yours, Roland
Hi Roland,
Thanks for your reply. You touch on an important point, i.e the cluster's billing policy, that hadn't crossed my mind.
From the billing policy of Leonardo, it seems that it is possible to use only a fraction of a node's total CPUs.
https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy
However, the documentation also stresses that: ...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.
So, apart from the cores requested, should I also try to calculate the RAM requirements? For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: memory = 494000 And the respective submitscript also has this line: #SBATCH --mem 494000MB
I note that each node in Leonardo has 512GB of RAM, so that means that the script requests ~94.2% of the RAM. I am not sure I follow the reasoning behind this.
What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think.
Best, Panayotis
________________________________ From: Roland Haas rhaas@mail.ubc.ca Sent: Friday, September 26, 2025 4:31 PM To: Bruno Giacomazzo bruno.giacomazzo@unimib.it Cc: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it; Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello all,
I never used --cores and I don't know the difference with procs.
--cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process).
Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hi all,
I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion.
1. Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html
2. Using 'leonardo-dcgp.ini' and understanding how to properly request one full node: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html
Thanks, Panayotis ________________________________ From: Users users-bounces@einsteintoolkit.org on behalf of IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it Sent: Monday, September 29, 2025 12:24 PM To: Roland Haas rhaas@mail.ubc.ca; Bruno Giacomazzo bruno.giacomazzo@unimib.it Cc: Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hi Roland,
Thanks for your reply. You touch on an important point, i.e the cluster's billing policy, that hadn't crossed my mind.
From the billing policy of Leonardo, it seems that it is possible to use only a fraction of a node's total CPUs.
https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy
However, the documentation also stresses that: ...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.
So, apart from the cores requested, should I also try to calculate the RAM requirements? For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: memory = 494000 And the respective submitscript also has this line: #SBATCH --mem 494000MB
I note that each node in Leonardo has 512GB of RAM, so that means that the script requests ~94.2% of the RAM. I am not sure I follow the reasoning behind this.
What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think.
Best, Panayotis
________________________________ From: Roland Haas rhaas@mail.ubc.ca Sent: Friday, September 26, 2025 4:31 PM To: Bruno Giacomazzo bruno.giacomazzo@unimib.it Cc: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it; Einstein Toolkit Users users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello all,
I never used --cores and I don't know the difference with procs.
--cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process).
Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote:
Hi all,
I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion.
Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html
Using 'leonardo-dcgp.ini' and understanding how to properly request one full node: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html
Normally, one requests --procs equal to the number of cores on the node.
So, imagine one has a machine with nodes that have 32 cores each.
One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8.
If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out.
--Steve
Thanks, Panayotis
*From:* Users users-bounces@einsteintoolkit.org on behalf of IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it *Sent:* Monday, September 29, 2025 12:24 PM *To:* Roland Haas rhaas@mail.ubc.ca; Bruno Giacomazzo bruno.giacomazzo@unimib.it *Cc:* Einstein Toolkit Users users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hi Roland,
Thanks for your reply. You touch on an important point, i.e the *cluster's* *billing policy*, that hadn't crossed my mind.
From the billing policy of Leonardo, it seems that *it is* *possible to use only a fraction of a node's total CPUs.* https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy
*However*, the documentation also stresses that: /...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.
/ So, apart from the cores requested, *should I also try to calculate the RAM requirements?* For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: |memory = 494000| And the respective submitscript also has this line: |#SBATCH --mem 494000MB|
I note that each node in Leonardo has 512GB of RAM, so that means that *the script requests ~94.2% of the RAM.* I am not sure I follow the reasoning behind this.
What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think.
Best, Panayotis
*From:* Roland Haas rhaas@mail.ubc.ca *Sent:* Friday, September 26, 2025 4:31 PM *To:* Bruno Giacomazzo bruno.giacomazzo@unimib.it *Cc:* IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it; Einstein Toolkit Users users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hello all,
I never used --cores and I don't know the difference with procs.
--cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process).
Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Dear Steve,
Thank you for your reply.
At this point, I reckon it makes sense to just use a full node (of 112 cores) for plain tests. In any case, they will finish quickly enough and not consume a lot of my allocated time.
I will leave the performance/convergence study varying the number of cores for the time being.
Regarding the --memory" option, I read the minutes from the last ETK meetinghttps://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html, and it seems that there is no obvious answer.
In case it is useful, I summarise my current understanding below:
* this wiki linkhttps://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine says that "memory is currently only used by simfactory's distribute utility script" * The distribute script seems to be a testing script only, so it does not seem to be relevant when we submit a job * There doesn't seem to be any source documenting exactly how simfactory handles memory * Without taking a look into the respective source code, my guess is that simfactory will try to use as much memory as needed by the job (but not necessarily the whole memory of the node) * In any case, if we request a full node, we will get billed accordingly * Only if we request less than a full node, would the --memory option possibly come into play and affect the billing
Best, Panagiotis
________________________________ From: Users users-bounces@einsteintoolkit.org on behalf of Steven Brandt via Users users@einsteintoolkit.org Sent: Thursday, October 23, 2025 4:28 PM To: users@einsteintoolkit.org users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote: Hi all,
I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion.
1. Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html
2. Using 'leonardo-dcgp.ini' and understanding how to properly request one full node: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html
Normally, one requests --procs equal to the number of cores on the node.
So, imagine one has a machine with nodes that have 32 cores each.
One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8.
If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out.
--Steve
1.
Thanks, Panayotis ________________________________ From: Users users-bounces@einsteintoolkit.orgmailto:users-bounces@einsteintoolkit.org on behalf of IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.itmailto:PANAGIOTIS.IOSIF@units.it Sent: Monday, September 29, 2025 12:24 PM To: Roland Haas rhaas@mail.ubc.camailto:rhaas@mail.ubc.ca; Bruno Giacomazzo bruno.giacomazzo@unimib.itmailto:bruno.giacomazzo@unimib.it Cc: Einstein Toolkit Users users@einsteintoolkit.orgmailto:users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hi Roland,
Thanks for your reply. You touch on an important point, i.e the cluster's billing policy, that hadn't crossed my mind.
From the billing policy of Leonardo, it seems that it is possible to use only a fraction of a node's total CPUs.
https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy
However, the documentation also stresses that: ...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.
So, apart from the cores requested, should I also try to calculate the RAM requirements? For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: memory = 494000 And the respective submitscript also has this line: #SBATCH --mem 494000MB
I note that each node in Leonardo has 512GB of RAM, so that means that the script requests ~94.2% of the RAM. I am not sure I follow the reasoning behind this.
What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think.
Best, Panayotis
________________________________ From: Roland Haas rhaas@mail.ubc.camailto:rhaas@mail.ubc.ca Sent: Friday, September 26, 2025 4:31 PM To: Bruno Giacomazzo bruno.giacomazzo@unimib.itmailto:bruno.giacomazzo@unimib.it Cc: IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.itmailto:PANAGIOTIS.IOSIF@units.it; Einstein Toolkit Users users@einsteintoolkit.orgmailto:users@einsteintoolkit.org Subject: Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
Hello all,
I never used --cores and I don't know the difference with procs.
--cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process).
Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
_______________________________________________ Users mailing list Users@einsteintoolkit.orgmailto:Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 10/24/2025 9:16 AM, IOSIF PANAGIOTIS wrote:
Dear Steve,
Thank you for your reply.
At this point, I reckon it makes sense to just use a full node (of 112 cores) for plain tests. In any case, they will finish quickly enough and not consume a lot of my allocated time.
Probably so. Maybe even a fraction of a node.
I will leave the performance/convergence study varying the number of cores for the time being.
Regarding the --memory" option, I read the minutes from the last ETK meeting https://lists.einsteintoolkit.org/pipermail/users/2025-October/009786.html, and it seems that there is no obvious answer.
Yes. No one seems to know. I suspect there's some way to pass it along to the submitscript where it can be given to slurm, but I'd have to try and read through the source code to figure that out.
--Steve
In case it is useful, I summarise my current understanding below:
- this wiki link https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine says that "|memory| is currently only used by simfactory's |distribute| utility script"
- The |distribute| script seems to be a testing script only, so it does not seem to be relevant when we submit a job
- There doesn't seem to be any source documenting exactly how simfactory handles memory
- Without taking a look into the respective source code, my guess is that simfactory will try to use as much memory as needed by the job (but not necessarily the whole memory of the node)
- In any case, if we request a full node, we will get billed accordingly
- /Only if we request less than a full node/, would the --memory option possibly come into play and affect the billing
Best, Panagiotis
*From:* Users users-bounces@einsteintoolkit.org on behalf of Steven Brandt via Users users@einsteintoolkit.org *Sent:* Thursday, October 23, 2025 4:28 PM *To:* users@einsteintoolkit.org users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster]
On 10/14/2025 3:55 AM, IOSIF PANAGIOTIS wrote:
Hi all,
I am sending a reminder regarding two unanswered questions on the mailing list, in case someone has a suggestion.
Clarification about how SimFactory handles the "--memory" option and how this affects how one should navigate the cluster's billing policy: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html https://lists.einsteintoolkit.org/pipermail/users/2025-September/009761.html
Using 'leonardo-dcgp.ini' and understanding how to properly request one full node: https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html https://lists.einsteintoolkit.org/pipermail/users/2025-September/009762.html
Normally, one requests --procs equal to the number of cores on the node.
So, imagine one has a machine with nodes that have 32 cores each.
One could say --procs 32, and that should be an entire node. However, maybe you want to run with 8 threads per MPI task. In that case, you would say --procs 32 --num-threads 8.
If you want to run on N nodes, then the number of procs would be 32*N, and Simfactory will figure it out.
--Steve
Thanks, Panayotis
*From:* Users users-bounces@einsteintoolkit.org mailto:users-bounces@einsteintoolkit.org on behalf of IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it mailto:PANAGIOTIS.IOSIF@units.it *Sent:* Monday, September 29, 2025 12:24 PM *To:* Roland Haas rhaas@mail.ubc.ca mailto:rhaas@mail.ubc.ca; Bruno Giacomazzo bruno.giacomazzo@unimib.it mailto:bruno.giacomazzo@unimib.it *Cc:* Einstein Toolkit Users users@einsteintoolkit.org mailto:users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hi Roland,
Thanks for your reply. You touch on an important point, i.e the *cluster's* *billing policy*, that hadn't crossed my mind.
From the billing policy of Leonardo, it seems that *it is* *possible to use only a fraction of a node's total CPUs.* https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy https://docs.hpc.cineca.it/hpc/hpc_intro.html#billing-policy
*However*, the documentation also stresses that: /...if a job reserves all of a node’s RAM — even without utilizing all its CPUs — the node becomes unusable for other jobs and is therefore billed accordingly.
/ So, apart from the cores requested, *should I also try to calculate the RAM requirements?* For example, I see that Bruno's "leonardo-dcgp.ini" file specifies: |memory = 494000| And the respective submitscript also has this line: |#SBATCH --mem 494000MB|
I note that each node in Leonardo has 512GB of RAM, so that means that *the script requests ~94.2% of the RAM.* I am not sure I follow the reasoning behind this.
What is the default behavior of SimFactory if I were to remove the above specifications from the config files? Because, if by default Simfactory requests/uses all the RAM available in a node, then as far as I understand, it does not make sense to request fewer cores than a full node. Let me know what you think.
Best, Panayotis
*From:* Roland Haas rhaas@mail.ubc.ca mailto:rhaas@mail.ubc.ca *Sent:* Friday, September 26, 2025 4:31 PM *To:* Bruno Giacomazzo bruno.giacomazzo@unimib.it mailto:bruno.giacomazzo@unimib.it *Cc:* IOSIF PANAGIOTIS PANAGIOTIS.IOSIF@units.it mailto:PANAGIOTIS.IOSIF@units.it; Einstein Toolkit Users users@einsteintoolkit.org mailto:users@einsteintoolkit.org *Subject:* Re: [Users] Inconsistency warnings: cores/threads mismatch [Leonardo cluster] Hello all,
I never used --cores and I don't know the difference with procs.
--cores is a synonym for --procs in simfactory. The hope was to avoid the confusion of "procs" being "Processes" or "Processors". Though it has been pointed out that the best name would actually be "--threads" since that is what simfactory actually starts, which then collides with "--num-threads" (threads per process).
Does Leonardo actually charge you for partial nodes if you do no use a full one? Simfactory is mostly written under the assumption (true at the time) that HPC systems would give you full nodes all the time, so if you use 1 core or 112 cores of a node, the charge would be the same (though shared node systems are becoming more common for HPC now [or again]).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu http://pgp.mit.edu .
Users mailing list Users@einsteintoolkit.org mailto:Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users http://lists.einsteintoolkit.org/mailman/listinfo/users
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Bruno Giacomazzo -
IOSIF PANAGIOTIS -
Roland Haas -
Steven Brandt