Hello Vaishak,
you options do not include optimization parameters ie:
CXX_OPTIMISE_FLAGS=
does not set any optimization options which means that g++ will compile as if -O0 was used.
Please change the option list that you are using to make sure that:
CXX_OPTIMISE_FLAGS=-O3 C_OPTIMISE_FLAGS=-O3 F90_OPTIMISE_FLAGS=-O3 F77_OPTIMISE_FLAGS=-O3
are set. If you are sure that login node and compute node are the same architecture you should also add "-march=native" to these options.
You may also consider using the Intel compiler (icc, icpc, ifort) instead of gcc (gcc, g++, gfortran) which may (or may not) give faster execution.
For example the file
https://bitbucket.org/simfactory/simfactory2/src/master/mdb/optionlists/stam...
shows the settings used to compile for the SkyLake nodes of the Stampede2 cluster at TACC using the Intel compiler.
I noticed that you are using a self-compiled MPI stack.
Usually on clusters the admins will provide an MPI stack optimized for the cluster hardware (eg to make sure Infiniband interconnects are used rather than Ethernet). I would expect better performance using those than using a self-compiled MPI stack (the Intel MPI stack will do in that respect since it is always provided by the admins and never self-compiled). Details on how to use it depend on the cluster and you would have to consult the cluster websites and / or the output of the cluster's mpicc -showme:compile and mpicc -showme:link commands (for OpenMPI mvapich and intel-mpi have similar options) to find out which libraries the official mpi compiler wrapper would use.
Yours, Roland
Dear Sir,
Please find the config-info file attached herewith...
Yours,
Vaishak
On Mon, Oct 7, 2019 at 7:56 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)?
Ideally if you could send the file configs/sim/config-info that would tell me.
Yours, Roland
Dear Sir,
I am a little worried about the performance because this is a new cluster we have and it is supposed to be performing well. I am inclined to think that some libraries/compiler options / settings might be the bottleneck.
I am presently running two simulations, both using the same parameter
file
GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation
mentioned
in the previous thread. I am using one node consisting of 2*16 cores, and 32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing computation. While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that one took about 137s for syncs and 198s for thorns, so the same ratio but about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI ranks can be detrimental if there is not enough work to split up (OpenMP can be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to run the simulation), can you try and see what would happen if you were to run on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would let us know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file
available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128
cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am
running
Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow
pace.
The number of physical time per hour that it is completing is only
about
1.3 units. At this rate to complete 1700 units, it would take about
54
days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available
at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .