<div dir="ltr"><div>Dear Sir,</div><div><br></div><div>I have made the changes as suggested. In fact I compiled using a Intel MPI ( which was compiled by myself locally) using the optionlist of Stanpede 2 cluster as suggested by you , without OpenMP and with appropriate library paths.I am glad that the speed has improved. I am now getting around 25 physical units per hour instead of 1.5 for a simulation running on 128 mpi procs. The optimization parameters I am using are same as in the Stanpede2 cluster ( -Ofast -march=native) and not "-O3 -march=native" . Would it make any difference?</div><div><br></div><div>I have not tried these changes on mpich-3.3.1 or openmpi yet. <br></div><div><br></div><div>Also where can I find more information to know about these optimization parameters? <br></div><div><br></div><div>Thankyou very much for your time. This was really helpful!!</div><div><br></div><div><br></div><div>Yours Sincerely</div><div><br></div><div>Vaishak <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 7, 2019 at 10:25 PM Haas, Roland <<a href="mailto:rhaas@illinois.edu">rhaas@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Vaishak,<br>
<br>
you options do not include optimization parameters ie:<br>
<br>
CXX_OPTIMISE_FLAGS=<br>
<br>
does not set any optimization options which means that g++ will compile<br>
as if -O0 was used.<br>
<br>
Please change the option list that you are using to make sure that:<br>
<br>
CXX_OPTIMISE_FLAGS=-O3<br>
C_OPTIMISE_FLAGS=-O3<br>
F90_OPTIMISE_FLAGS=-O3<br>
F77_OPTIMISE_FLAGS=-O3<br>
<br>
are set. If you are sure that login node and compute node are the same<br>
architecture you should also add "-march=native" to these options.<br>
<br>
You may also consider using the Intel compiler (icc, icpc,<br>
ifort) instead of gcc (gcc, g++, gfortran) which may (or may not) give<br>
faster execution.<br>
<br>
For example the file <br>
<br>
<a href="https://bitbucket.org/simfactory/simfactory2/src/master/mdb/optionlists/stampede2-skx.cfg" rel="noreferrer" target="_blank">https://bitbucket.org/simfactory/simfactory2/src/master/mdb/optionlists/stampede2-skx.cfg</a><br>
<br>
shows the settings used to compile for the SkyLake nodes of the<br>
Stampede2 cluster at TACC using the Intel compiler.<br>
<br>
I noticed that you are using a self-compiled MPI stack.<br>
<br>
Usually on clusters the admins will provide an MPI stack optimized for<br>
the cluster hardware (eg to make sure Infiniband interconnects are used<br>
rather than Ethernet). I would expect better performance using those<br>
than using a self-compiled MPI stack (the Intel MPI stack will do in<br>
that respect since it is always provided by the admins and never<br>
self-compiled). Details on how to use it depend on the cluster and you<br>
would have to consult the cluster websites and / or the output of the<br>
cluster's mpicc -showme:compile and mpicc -showme:link commands (for<br>
OpenMPI mvapich and intel-mpi have similar options) to find out which<br>
libraries the official mpi compiler wrapper would use.<br>
<br>
Yours,<br>
Roland<br>
<br>
> Dear Sir,<br>
> <br>
> Please find the config-info file attached herewith...<br>
> <br>
> <br>
> Yours,<br>
> <br>
> Vaishak<br>
> <br>
> On Mon, Oct 7, 2019 at 7:56 PM Haas, Roland <<a href="mailto:rhaas@illinois.edu" target="_blank">rhaas@illinois.edu</a>> wrote:<br>
> <br>
> > Hello Vaishak,<br>
> ><br>
> > hmm, still very slow.<br>
> ><br>
> > One question that I forgot to ask before: did you make sure to build<br>
> > and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to<br>
> > ensure that you have -O2 or -O3 optimisation settings enabled)?<br>
> ><br>
> > Ideally if you could send the file configs/sim/config-info that would<br>
> > tell me.<br>
> ><br>
> > Yours,<br>
> > Roland<br>
> ><br>
> > > Dear Sir,<br>
> > ><br>
> > > I am a little worried about the performance because this is a new cluster<br>
> > > we have and it is supposed to be performing well. I am inclined to think<br>
> > > that some libraries/compiler options / settings might be the bottleneck.<br>
> > ><br>
> > ><br>
> > > I am presently running two simulations, both using the same parameter<br>
> > file<br>
> > > GW150914.rpar.<br>
> > ><br>
> > > The first one is using mpich-3.3.1, the same as in the simulation<br>
> > mentioned<br>
> > > in the previous thread. I am using one node consisting of 2*16 cores, and<br>
> > > 32 mpiprocs.<br>
> > ><br>
> > > The second one is using openmpi-3.1.2 with openmp. It uses 128 procs,<br>
> > > distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I<br>
> > > have 32 PPN, it is launching 4 mpiprocs per node.<br>
> > ><br>
> > > I am herewith attaching the carpet-timing..asc file from both these runs.<br>
> > ><br>
> > > Thanking you<br>
> > ><br>
> > > Regards,<br>
> > > Vaishak<br>
> > ><br>
> > ><br>
> > > On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland <<a href="mailto:rhaas@illinois.edu" target="_blank">rhaas@illinois.edu</a>> wrote:<br>
> > ><br>
> > > > Hello Vaishak,<br>
> > > ><br>
> > > > I do not see anything obviously wrong with the setup.<br>
> > > ><br>
> > > > It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16<br>
> > > > cores per node.<br>
> > > ><br>
> > > > Lookin at the timer tree output at iteration 1024 (search for<br>
> > > > "gettimeof " and you will find the spot) out of 5977s spend during<br>
> > > > Evolve about 2143s were spent in "syncs" which is communication and<br>
> > > > about the same amount of time in "thorns" that is doing computation.<br>
> > > > While this ratio is not great (spending more time sending data than<br>
> > > > doing computation) it is also not unheard of.<br>
> > > ><br>
> > > > Getting the original output files for the gallery data from Zenodo<br>
> > > > (link is on the gallery page):<br>
> > > ><br>
> > > > wget<br>
> > > > <a href="https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1" rel="noreferrer" target="_blank">https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1</a><br>
> > > ><br>
> > > > you can see (in GW150914_28/output-0000/GW150914_28.out) that that one<br>
> > > > took about 137s for syncs and 198s for thorns, so the same ratio but<br>
> > > > about a factor of 10 faster.<br>
> > > ><br>
> > > > I am reaching for straws here, but sometimes having too many MPI ranks<br>
> > > > can be detrimental if there is not enough work to split up (OpenMP can<br>
> > > > be a bit more forgiving in that respect, the original gallery run<br>
> > > > used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).<br>
> > > ><br>
> > > > Since each node has lots of RAM (more than the 96GB required to run the<br>
> > > > simulation), can you try and see what would happen if you were to run<br>
> > > > on only a single node?<br>
> > > ><br>
> > > > Also if you could add the parameter:<br>
> > > ><br>
> > > > Carpet::output_timers_every = 1024<br>
> > > ><br>
> > > > then provide the files carpet-timing-statistics*.asc that would let us<br>
> > > > know in even more detail where the time is spent.<br>
> > > ><br>
> > > > Running for a short time (2048 iterations) is enough to get data to<br>
> > > > compare.<br>
> > > ><br>
> > > > Yours,<br>
> > > > Roland<br>
> > > ><br>
> > > > > Dear All,<br>
> > > > ><br>
> > > > > I am running the simulation GW150914 using the parameter file<br>
> > available<br>
> > > > at<br>
> > > > > the ETK gallery at (GW150914-ETK gallery<br>
> > > > > <<a href="https://einsteintoolkit.org/gallery/bbh/index.html" rel="noreferrer" target="_blank">https://einsteintoolkit.org/gallery/bbh/index.html</a>>) using 128<br>
> > cores.<br>
> > > > ><br>
> > > > > Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)<br>
> > > > Xeon(R)<br>
> > > > > Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am<br>
> > running<br>
> > > > > Einstein Toolkit without OpenMP and using mpich-3.3.1.<br>
> > > > ><br>
> > > > ><br>
> > > > > The issue is that the simulation seems to be running at a very slow<br>
> > pace.<br>
> > > > > The number of physical time per hour that it is completing is only<br>
> > about<br>
> > > > > 1.3 units. At this rate to complete 1700 units, it would take about<br>
> > 54<br>
> > > > > days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @<br>
> > > > > 2.40GHz) as per the details at the example run of GW150914 available<br>
> > at<br>
> > > > the<br>
> > > > > gallery (GW150914-ETK gallery<br>
> > > > > <<a href="https://einsteintoolkit.org/gallery/bbh/index.html" rel="noreferrer" target="_blank">https://einsteintoolkit.org/gallery/bbh/index.html</a>>).<br>
> > > > ><br>
> > > > > I have also tried using intel mpi (impi) but with simular results.<br>
> > > > ><br>
> > > > > I am also attaching the out file from the simulation.<br>
> > > > ><br>
> > > > > Looking forward to your inputs.<br>
> > > > ><br>
> > > > ><br>
> > > > > Thanks and regards,<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > Vaishak P<br>
> > > > ><br>
> > > > > PhD Scholar,<br>
> > > > > Shyama Prasad Mukherjee Fellow<br>
> > > > > Inter-University Center for Astronomy and Astrophysics (IUCAA)<br>
> > > > > Pune, India<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > My email is as private as my paper mail. I therefore support encrypting<br>
> > > > and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
> > > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > My email is as private as my paper mail. I therefore support encrypting<br>
> > and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
> ><br>
> <br>
> <br>
<br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Regards,<br>Vaishak P<br><br>PhD Scholar,<br>Shyama Prasad Mukherjee Fellow<br>
Inter-University Center for Astronomy and Astrophysics (IUCAA)<br>
Pune, India<div><br></div></div></div>