Dear Sir,
I have made the changes as suggested. In fact I compiled using a Intel MPI ( which was compiled by myself locally) using the optionlist of Stanpede 2 cluster as suggested by you , without OpenMP and with appropriate library paths.I am glad that the speed has improved. I am now getting around 25 physical units per hour instead of 1.5 for a simulation running on 128 mpi procs. The optimization parameters I am using are same as in the Stanpede2 cluster ( -Ofast -march=native) and not "-O3 -march=native" . Would it make any difference?
I have not tried these changes on mpich-3.3.1 or openmpi yet.
Also where can I find more information to know about these optimization parameters?
Thankyou very much for your time. This was really helpful!!
Yours Sincerely
Vaishak
On Mon, Oct 7, 2019 at 10:25 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
you options do not include optimization parameters ie:
CXX_OPTIMISE_FLAGS=
does not set any optimization options which means that g++ will compile as if -O0 was used.
Please change the option list that you are using to make sure that:
CXX_OPTIMISE_FLAGS=-O3 C_OPTIMISE_FLAGS=-O3 F90_OPTIMISE_FLAGS=-O3 F77_OPTIMISE_FLAGS=-O3
are set. If you are sure that login node and compute node are the same architecture you should also add "-march=native" to these options.
You may also consider using the Intel compiler (icc, icpc, ifort) instead of gcc (gcc, g++, gfortran) which may (or may not) give faster execution.
For example the file
https://bitbucket.org/simfactory/simfactory2/src/master/mdb/optionlists/stam...
shows the settings used to compile for the SkyLake nodes of the Stampede2 cluster at TACC using the Intel compiler.
I noticed that you are using a self-compiled MPI stack.
Usually on clusters the admins will provide an MPI stack optimized for the cluster hardware (eg to make sure Infiniband interconnects are used rather than Ethernet). I would expect better performance using those than using a self-compiled MPI stack (the Intel MPI stack will do in that respect since it is always provided by the admins and never self-compiled). Details on how to use it depend on the cluster and you would have to consult the cluster websites and / or the output of the cluster's mpicc -showme:compile and mpicc -showme:link commands (for OpenMPI mvapich and intel-mpi have similar options) to find out which libraries the official mpi compiler wrapper would use.
Yours, Roland
Dear Sir,
Please find the config-info file attached herewith...
Yours,
Vaishak
On Mon, Oct 7, 2019 at 7:56 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)?
Ideally if you could send the file configs/sim/config-info that would tell me.
Yours, Roland
Dear Sir,
I am a little worried about the performance because this is a new
cluster
we have and it is supposed to be performing well. I am inclined to
think
that some libraries/compiler options / settings might be the
bottleneck.
I am presently running two simulations, both using the same parameter
file
GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation
mentioned
in the previous thread. I am using one node consisting of 2*16
cores, and
32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc.
Since I
have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these
runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu
wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being
2x16
cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing
computation.
While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget
https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that
one
took about 137s for syncs and 198s for thorns, so the same ratio
but
about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI
ranks
can be detrimental if there is not enough work to split up (OpenMP
can
be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to
run the
simulation), can you try and see what would happen if you were to
run
on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would
let us
know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file
available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128
cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake (
Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am
running
Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very
slow
pace.
The number of physical time per hour that it is completing is
only
about
1.3 units. At this rate to complete 1700 units, it would take
about
54
days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630
v3 @
2.40GHz) as per the details at the example run of GW150914
available
at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular
results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support
encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu
.
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .