Hello,
I've downloaded and configured the Einstein Toolkit on my desktop, but I'm having an issue when I try to run the BBH example provided here https://einsteintoolkit.org/gallery/bbh/index.html and the BNS simulation from here https://einsteintoolkit.org/gallery/bns/index.html. The job finishes in an instant, producing the attached error file. (I'm using the command "simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 24 --walltime 24:00:00")
There seems to be an issue with the 'num_pus' to 'num_cores' ratio in the "system_topology.cc" file, but the error oddly persists even when I manually equate them, or cut out line 471 altogether. This error does not appear when I run the tov_ET example or the HelloWorld one. I'm also able to run this mini BNS tutorial https://www.youtube.com/watch?v=3euS5yMBFmI without any issues, so I'm a bit baffled. I've attached my .ini file and the .cc file along with the error file in case it helps. Please take a look and let me know if I'm missing something obvious.
PS: My PC has 128GB of memory, a 13900K(24C/32T) processor and 1TB of storage.
Regards, Praveer
Hello Krishna,
The abort is triggered by the SystemTopology / HWLOC thorns having encountered a processor / core / PU combination that they did not expect. Is there anything in the stdout file?
I am not quite familiar enough with hwloc in particular to really make a good suggestion.
You can certainly try and remove
SystemTopology
and
hwloc
from your parameter file's ActiveThorn line(s). This will avoid the issue but may, depending on your cluster (mis-)configuration, yield non-ideal speeds (basically you must ensure manually that there is at most one thread per CPU core). Carpet has an option
Carpet::set_cpu_affinity = "yes"
that probably gives you almost all you would get from SystemTopology so you can try that one instead.
Can you provide details on the cluster / CPU of the cluster?
Yours, Roland
Hello,
I've downloaded and configured the Einstein Toolkit on my desktop, but I'm having an issue when I try to run the BBH example provided here <https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bbh/index.ht...
and the BNS simulation from here <https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bns/index.ht...
. The job finishes in an instant, producing the attached error file. (I'm using the
command "simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 24 --walltime 24:00:00")
There seems to be an issue with the 'num_pus' to 'num_cores' ratio in the "system_topology.cc" file, but the error oddly persists even when I manually equate them, or cut out line 471 altogether. This error does not appear when I run the tov_ET example or the HelloWorld one. I'm also able to run this mini BNS tutorial <https://urldefense.com/v3/__https://www.youtube.com/watch?v=3euS5yMBFmI__;!!...
without any issues, so I'm a bit baffled. I've attached my .ini file and the .cc file along with the error file in case it helps.
Please take a look and let me know if I'm missing something obvious.
PS: My PC has 128GB of memory, a 13900K(24C/32T) processor and 1TB of storage.
Regards, Praveer
Hello Krishna,
Turns out I am getting a similar issue myself now when running on my (old, well tested) laptop:
cactus_sim: configs/sim/build/SystemTopology/system_topology.cc:471: void {anonymous}::set_bindings(hwloc_topology_t, const mpi_host_mapping_t&): Assertion `num_pus % num_cores == 0' failed.
so something may have changed in hwloc even for standard machines. I will see if this is something that can be fixed in code.
Yours, Roland
Hello Krishna,
The abort is triggered by the SystemTopology / HWLOC thorns having encountered a processor / core / PU combination that they did not expect. Is there anything in the stdout file?
I am not quite familiar enough with hwloc in particular to really make a good suggestion.
You can certainly try and remove
SystemTopology
and
hwloc
from your parameter file's ActiveThorn line(s). This will avoid the issue but may, depending on your cluster (mis-)configuration, yield non-ideal speeds (basically you must ensure manually that there is at most one thread per CPU core). Carpet has an option
Carpet::set_cpu_affinity = "yes"
that probably gives you almost all you would get from SystemTopology so you can try that one instead.
Can you provide details on the cluster / CPU of the cluster?
Yours, Roland
Hello,
I've downloaded and configured the Einstein Toolkit on my desktop, but I'm having an issue when I try to run the BBH example provided here <https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bbh/index.ht...
and the BNS simulation from here <https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bns/index.ht...
. The job finishes in an instant, producing the attached error file. (I'm using the
command "simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 24 --walltime 24:00:00")
There seems to be an issue with the 'num_pus' to 'num_cores' ratio in the "system_topology.cc" file, but the error oddly persists even when I manually equate them, or cut out line 471 altogether. This error does not appear when I run the tov_ET example or the HelloWorld one. I'm also able to run this mini BNS tutorial <https://urldefense.com/v3/__https://www.youtube.com/watch?v=3euS5yMBFmI__;!!...
without any issues, so I'm a bit baffled. I've attached my .ini file and the .cc file along with the error file in case it helps.
Please take a look and let me know if I'm missing something obvious.
PS: My PC has 128GB of memory, a 13900K(24C/32T) processor and 1TB of storage.
Regards, Praveer
Hello Krishna, all,
Correction. This is happening on my brand new workstation with
12th Gen Intel(R) Core(TM) i7-12700
https://en.wikipedia.org/wiki/List_of_Intel_Core_i7_processors#Golden_Cove_+...)
with performance (16) and efficiency (4) logical cpus.
With this hwloc reports:
core_depth 5 num_cores 12 pu_depth 6 num_pus 20
and the assert num_pus % num_cores fails (b/c only the 8 performance cores have hyperthreads).
So an newly developed issue in systemtopology's assumptions due to changing hardware, really.
I have created a ticket for this (https://bitbucket.org/einsteintoolkit/tickets/issues/2756) since this needs to be fixed since one would expect the new Intel CPUs with mixed cores to become more common at least on laptops and workstations.
Yours, Roland
Hello Krishna,
Turns out I am getting a similar issue myself now when running on my (old, well tested) laptop:
cactus_sim: configs/sim/build/SystemTopology/system_topology.cc:471: void {anonymous}::set_bindings(hwloc_topology_t, const mpi_host_mapping_t&): Assertion `num_pus % num_cores == 0' failed.
so something may have changed in hwloc even for standard machines. I will see if this is something that can be fixed in code.
Yours, Roland
Hello Krishna,
The abort is triggered by the SystemTopology / HWLOC thorns having encountered a processor / core / PU combination that they did not expect. Is there anything in the stdout file?
I am not quite familiar enough with hwloc in particular to really make a good suggestion.
You can certainly try and remove
SystemTopology
and
hwloc
from your parameter file's ActiveThorn line(s). This will avoid the issue but may, depending on your cluster (mis-)configuration, yield non-ideal speeds (basically you must ensure manually that there is at most one thread per CPU core). Carpet has an option
Carpet::set_cpu_affinity = "yes"
that probably gives you almost all you would get from SystemTopology so you can try that one instead.
Can you provide details on the cluster / CPU of the cluster?
Yours, Roland
Hello,
I've downloaded and configured the Einstein Toolkit on my desktop, but I'm having an issue when I try to run the BBH example provided here <https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bbh/index.ht...
and the BNS simulation from here <https://urldefense.com/v3/__https://einsteintoolkit.org/gallery/bns/index.ht...
. The job finishes in an instant, producing the attached error file. (I'm using the
command "simfactory/bin/sim create-submit GW150914_28 --define N 28 --parfile par/GW150914/GW150914.rpar --procs 24 --walltime 24:00:00")
There seems to be an issue with the 'num_pus' to 'num_cores' ratio in the "system_topology.cc" file, but the error oddly persists even when I manually equate them, or cut out line 471 altogether. This error does not appear when I run the tov_ET example or the HelloWorld one. I'm also able to run this mini BNS tutorial <https://urldefense.com/v3/__https://www.youtube.com/watch?v=3euS5yMBFmI__;!!...
without any issues, so I'm a bit baffled. I've attached my .ini file and the .cc file along with the error file in case it helps.
Please take a look and let me know if I'm missing something obvious.
PS: My PC has 128GB of memory, a 13900K(24C/32T) processor and 1TB of storage.
Regards, Praveer
users@lists.einsteintoolkit.org