Thanks Ian, I think that did the trick! I rebuilt Cactus with a different MPI infrastructure and made sure that the appropriate MPI environment was loaded. Seems to be working now, e.g.
INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 12 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 1 threads, this is thread 0
INFO (Carpet): There are 12 threads in total
INFO (Carpet): There are 1 threads per process
INFO (Carpet): This process runs on host node203, pid=28712
INFO (Carpet): This process runs on 24 cores: 0-23
INFO (Carpet): Thread 0 runs on 24 cores: 0-23
Thanks! Geraint
On 20 October 2015 at 14:55, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 20 Oct 2015, at 15:44, Geraint Pratten g.pratten@sussex.ac.uk wrote:
Thanks! I'll take a look into that - changing to $MPIDIR/bin/mpirun didn't help but I suspect that the MPI version used in the options list may not be correct. Will see if this is the issue and confirm.
You could also check to see which MPI module is loaded. Do you load any modules in envsetup in the machine definition file?
-- Ian Hinder http://members.aei.mpg.de/ianhin
On 20 Oct 2015, at 22:13, Geraint Pratten g.pratten@sussex.ac.uk wrote:
Thanks Ian, I think that did the trick! I rebuilt Cactus with a different MPI infrastructure and made sure that the appropriate MPI environment was loaded. Seems to be working now, e.g.
INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 12 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 1 threads, this is thread 0 INFO (Carpet): There are 12 threads in total INFO (Carpet): There are 1 threads per process INFO (Carpet): This process runs on host node203, pid=28712 INFO (Carpet): This process runs on 24 cores: 0-23 INFO (Carpet): Thread 0 runs on 24 cores: 0-23
Thanks! Geraint
Glad to hear it's working!
The last line indicates that the threads are not being pinned to specific cores. Usually, we recommend to pin them. You should be able to do this by activating thorns hwloc and SystemTopology in your parameter file.
Note to others: with a change in the last few months, some of the functionality of hwloc was moved into SystemTopology, which is not automatically activated. If you want to get automatic thread pinning, you now need to edit your parameter files to explicitly activate SystemTopology in addition to hwloc.
Erik, could SystemTopology be automatically activated somehow?
Hello all,
Glad to hear it's working!
The last line indicates that the threads are not being pinned to specific cores. Usually, we recommend to pin them. You should be able to do this by activating thorns hwloc and SystemTopology in your parameter file.
I suggest to also use your mpirun command's pinning options if available. This can be an issue since MPI will allocate memory before thorns hwloc and SystemTopology have a change of pinning the process so that you may end up with cores having to use memory that is physically attached to the "other" sockets in multi-socket systems.
There is unfortunately nothing any thorn can do about this since MPI may allocate memory as soon as MPI_Init is called by the flesh.
Yours, Roland
On Wed, Oct 21, 2015 at 3:49 AM, Roland Haas rhaas@aei.mpg.de wrote:
Hello all,
Glad to hear it's working!
The last line indicates that the threads are not being pinned to specific cores. Usually, we recommend to pin them. You should be able to do this by activating thorns hwloc and SystemTopology in your parameter file.
I suggest to also use your mpirun command's pinning options if available. This can be an issue since MPI will allocate memory before thorns hwloc and SystemTopology have a change of pinning the process so that you may end up with cores having to use memory that is physically attached to the "other" sockets in multi-socket systems.
There is unfortunately nothing any thorn can do about this since MPI may allocate memory as soon as MPI_Init is called by the flesh.
We can introduce a mechanism by which a thorn can be called before MPI_Init is called. This would call all thorns that are compiled in, since the parameter file has not been read yet.
-erik
On Wed, Oct 21, 2015 at 3:25 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 20 Oct 2015, at 22:13, Geraint Pratten g.pratten@sussex.ac.uk wrote:
Thanks Ian, I think that did the trick! I rebuilt Cactus with a different MPI infrastructure and made sure that the appropriate MPI environment was loaded. Seems to be working now, e.g.
INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 12 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 1 threads, this is thread 0 INFO (Carpet): There are 12 threads in total INFO (Carpet): There are 1 threads per process INFO (Carpet): This process runs on host node203, pid=28712 INFO (Carpet): This process runs on 24 cores: 0-23 INFO (Carpet): Thread 0 runs on 24 cores: 0-23
Thanks! Geraint
Glad to hear it's working!
The last line indicates that the threads are not being pinned to specific cores. Usually, we recommend to pin them. You should be able to do this by activating thorns hwloc and SystemTopology in your parameter file.
Note to others: with a change in the last few months, some of the functionality of hwloc was moved into SystemTopology, which is not automatically activated. If you want to get automatic thread pinning, you now need to edit your parameter files to explicitly activate SystemTopology in addition to hwloc.
Erik, could SystemTopology be automatically activated somehow?
Ian
Yes, the way to do this is to have another thorn (that is already activated) optionally require this thorn. Obvious choices are the flesh, or Carpet (if you don't care about PUGH users).
-erik
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Geraint Pratten -
Ian Hinder -
Roland Haas