Hello ZhiChao,

sorry for the late reply. My best guess is that you are oversubscribing the compute core of your machine.

By default (ie when no OMP_NUM_THREADS) variable is set, OpenMP will use all cores of a machine. So if you start the simulation with mpirun -n2 you will end up with 2 processes (MPI ranks) *each* of which spawns as many threads as there are cores on your machine. This makes things very slow due to scheduling overhead etc.

You can check this (on Linux) by having top show you the individual threads, eg by calling it like so:

top -H

You can also try if:

export OMP_NUM_THREADS=3 mpirun -np 2 ./cactus_sim nsns.par

helps (assuming you have 6 cores in total on your machine)

Yours, Roland

Hello, everyone,

I am a beginner in Institute of High Energy Physics, China.

I am using ubuntu16.04 and my machine has two cpus:

   Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz X2

I am trying to repeat "Gallery: Binary, inspiraling neutron stars forming a hypermassive neutron star" in http://einsteintoolkit.org/gallery/bns/index.html. I am using the par file and thornlist.th provided in the webpage.

However, I found it runs slowly when I run with "mpirun -np 2 ./Cactus_sim nsns.par". It needs about 600s per iteration. All cpu's usage are 100%.

Then I try to directly run with command "./Cactus_sim nsns.par", then it needs about 13s per iteration. All cpu's usage are 100%.

And then, I try to run with "numactl --cpunodebind=0 --preferred=0 ./cactus_sim nsns.par", It only needs 2.5s per iteration. And just half cpu ( 28 of 56 )'s usage are 100%.

I wonder is there some error? I think it should be faster when mpirun are used.

Yours, ZhiChao.