[Users] Using "mpirun -np 2" make the program much slower ?

[Roland Haas]

Hello ZhiChao,

sorry for the late reply. My best guess is that you are oversubscribing
the compute core of your machine. 

By default (ie when no OMP_NUM_THREADS) variable is set, OpenMP will
use all cores of a machine. So if you start the simulation with mpirun
-n2 you will end up with 2 processes (MPI ranks) *each* of which spawns
as many threads as there are cores on your machine. This makes things
very slow due to scheduling overhead etc. 

You can check this (on Linux) by having top show you the individual
threads, eg by calling it like so:

top -H

You can also try if:

export OMP_NUM_THREADS=3
mpirun -np 2 ./cactus_sim nsns.par

helps (assuming you have 6 cores in total on your machine) 

Yours,
Roland

> Hello, everyone,
> 
> I am a beginner in Institute of High Energy Physics, China.
> 
> I am using ubuntu16.04 and my machine has two cpus:
> 
>        Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz X2
> 
> I am trying to repeat "Gallery: Binary, inspiraling neutron stars forming a
> hypermassive neutron star" in
> http://einsteintoolkit.org/gallery/bns/index.html. I am using the par file
> and thornlist.th provided in the webpage.
> 
> However, I found it runs slowly when I run with "mpirun -np 2 ./Cactus_sim
> nsns.par". It needs about 600s per iteration. All cpu's usage are 100%.
> 
> Then I try to directly run with command "./Cactus_sim nsns.par", then it
> needs about 13s per iteration. All cpu's usage are 100%.
> 
> And then, I try to run with "numactl --cpunodebind=0 --preferred=0
> ./cactus_sim nsns.par", It only needs 2.5s per iteration. And just half cpu
> ( 28 of 56 )'s usage are 100%.
> 
> I wonder is there some error? I think it should be faster when mpirun are
> used.
> 
> Yours,
> ZhiChao.



-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20170423/628d44a8/attachment.bin