On 20 Oct 2015, at 22:13, Geraint Pratten <g.pratten@sussex.ac.uk> wrote:

Thanks Ian, I think that did the trick! I rebuilt Cactus with a different MPI infrastructure and made sure that the appropriate MPI environment was loaded. Seems to be working now, e.g.

INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 12 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 1 threads, this is thread 0
INFO (Carpet): There are 12 threads in total
INFO (Carpet): There are 1 threads per process
INFO (Carpet): This process runs on host node203, pid=28712
INFO (Carpet): This process runs on 24 cores: 0-23
INFO (Carpet): Thread 0 runs on 24 cores: 0-23

Thanks!
Geraint

Glad to hear it's working!

The last line indicates that the threads are not being pinned to specific cores. Usually, we recommend to pin them. You should be able to do this by activating thorns hwloc and SystemTopology in your parameter file.

Note to others: with a change in the last few months, some of the functionality of hwloc was moved into SystemTopology, which is not automatically activated. If you want to get automatic thread pinning, you now need to edit your parameter files to explicitly activate SystemTopology in addition to hwloc.

Erik, could SystemTopology be automatically activated somehow?