Dear All,
I am running the simulation GW150914 using the parameter file available at the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128 cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am running Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow pace. The number of physical time per hour that it is completing is only about 1.3 units. At this rate to complete 1700 units, it would take about 54 days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available at the gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing computation. While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that one took about 137s for syncs and 198s for thorns, so the same ratio but about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI ranks can be detrimental if there is not enough work to split up (OpenMP can be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to run the simulation), can you try and see what would happen if you were to run on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would let us know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file available at the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128 cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am running Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow pace. The number of physical time per hour that it is completing is only about 1.3 units. At this rate to complete 1700 units, it would take about 54 days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available at the gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
Dear Sir,
I am a little worried about the performance because this is a new cluster we have and it is supposed to be performing well. I am inclined to think that some libraries/compiler options / settings might be the bottleneck.
I am presently running two simulations, both using the same parameter file GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation mentioned in the previous thread. I am using one node consisting of 2*16 cores, and 32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing computation. While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that one took about 137s for syncs and 198s for thorns, so the same ratio but about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI ranks can be detrimental if there is not enough work to split up (OpenMP can be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to run the simulation), can you try and see what would happen if you were to run on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would let us know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128 cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am running Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow pace. The number of physical time per hour that it is completing is only about 1.3 units. At this rate to complete 1700 units, it would take about 54 days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Vaishak,
hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)?
Ideally if you could send the file configs/sim/config-info that would tell me.
Yours, Roland
Dear Sir,
I am a little worried about the performance because this is a new cluster we have and it is supposed to be performing well. I am inclined to think that some libraries/compiler options / settings might be the bottleneck.
I am presently running two simulations, both using the same parameter file GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation mentioned in the previous thread. I am using one node consisting of 2*16 cores, and 32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing computation. While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that one took about 137s for syncs and 198s for thorns, so the same ratio but about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI ranks can be detrimental if there is not enough work to split up (OpenMP can be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to run the simulation), can you try and see what would happen if you were to run on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would let us know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128 cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am running Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow pace. The number of physical time per hour that it is completing is only about 1.3 units. At this rate to complete 1700 units, it would take about 54 days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Dear Sir,
Please find the config-info file attached herewith...
Yours,
Vaishak
On Mon, Oct 7, 2019 at 7:56 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)?
Ideally if you could send the file configs/sim/config-info that would tell me.
Yours, Roland
Dear Sir,
I am a little worried about the performance because this is a new cluster we have and it is supposed to be performing well. I am inclined to think that some libraries/compiler options / settings might be the bottleneck.
I am presently running two simulations, both using the same parameter
file
GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation
mentioned
in the previous thread. I am using one node consisting of 2*16 cores, and 32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing computation. While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that one took about 137s for syncs and 198s for thorns, so the same ratio but about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI ranks can be detrimental if there is not enough work to split up (OpenMP can be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to run the simulation), can you try and see what would happen if you were to run on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would let us know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file
available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128
cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am
running
Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow
pace.
The number of physical time per hour that it is completing is only
about
1.3 units. At this rate to complete 1700 units, it would take about
54
days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available
at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Vaishak,
you options do not include optimization parameters ie:
CXX_OPTIMISE_FLAGS=
does not set any optimization options which means that g++ will compile as if -O0 was used.
Please change the option list that you are using to make sure that:
CXX_OPTIMISE_FLAGS=-O3 C_OPTIMISE_FLAGS=-O3 F90_OPTIMISE_FLAGS=-O3 F77_OPTIMISE_FLAGS=-O3
are set. If you are sure that login node and compute node are the same architecture you should also add "-march=native" to these options.
You may also consider using the Intel compiler (icc, icpc, ifort) instead of gcc (gcc, g++, gfortran) which may (or may not) give faster execution.
For example the file
https://bitbucket.org/simfactory/simfactory2/src/master/mdb/optionlists/stam...
shows the settings used to compile for the SkyLake nodes of the Stampede2 cluster at TACC using the Intel compiler.
I noticed that you are using a self-compiled MPI stack.
Usually on clusters the admins will provide an MPI stack optimized for the cluster hardware (eg to make sure Infiniband interconnects are used rather than Ethernet). I would expect better performance using those than using a self-compiled MPI stack (the Intel MPI stack will do in that respect since it is always provided by the admins and never self-compiled). Details on how to use it depend on the cluster and you would have to consult the cluster websites and / or the output of the cluster's mpicc -showme:compile and mpicc -showme:link commands (for OpenMPI mvapich and intel-mpi have similar options) to find out which libraries the official mpi compiler wrapper would use.
Yours, Roland
Dear Sir,
Please find the config-info file attached herewith...
Yours,
Vaishak
On Mon, Oct 7, 2019 at 7:56 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)?
Ideally if you could send the file configs/sim/config-info that would tell me.
Yours, Roland
Dear Sir,
I am a little worried about the performance because this is a new cluster we have and it is supposed to be performing well. I am inclined to think that some libraries/compiler options / settings might be the bottleneck.
I am presently running two simulations, both using the same parameter
file
GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation
mentioned
in the previous thread. I am using one node consisting of 2*16 cores, and 32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16 cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing computation. While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that one took about 137s for syncs and 198s for thorns, so the same ratio but about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI ranks can be detrimental if there is not enough work to split up (OpenMP can be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to run the simulation), can you try and see what would happen if you were to run on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would let us know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file
available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128
cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am
running
Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very slow
pace.
The number of physical time per hour that it is completing is only
about
1.3 units. At this rate to complete 1700 units, it would take about
54
days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz) as per the details at the example run of GW150914 available
at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Dear Sir,
I have made the changes as suggested. In fact I compiled using a Intel MPI ( which was compiled by myself locally) using the optionlist of Stanpede 2 cluster as suggested by you , without OpenMP and with appropriate library paths.I am glad that the speed has improved. I am now getting around 25 physical units per hour instead of 1.5 for a simulation running on 128 mpi procs. The optimization parameters I am using are same as in the Stanpede2 cluster ( -Ofast -march=native) and not "-O3 -march=native" . Would it make any difference?
I have not tried these changes on mpich-3.3.1 or openmpi yet.
Also where can I find more information to know about these optimization parameters?
Thankyou very much for your time. This was really helpful!!
Yours Sincerely
Vaishak
On Mon, Oct 7, 2019 at 10:25 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
you options do not include optimization parameters ie:
CXX_OPTIMISE_FLAGS=
does not set any optimization options which means that g++ will compile as if -O0 was used.
Please change the option list that you are using to make sure that:
CXX_OPTIMISE_FLAGS=-O3 C_OPTIMISE_FLAGS=-O3 F90_OPTIMISE_FLAGS=-O3 F77_OPTIMISE_FLAGS=-O3
are set. If you are sure that login node and compute node are the same architecture you should also add "-march=native" to these options.
You may also consider using the Intel compiler (icc, icpc, ifort) instead of gcc (gcc, g++, gfortran) which may (or may not) give faster execution.
For example the file
https://bitbucket.org/simfactory/simfactory2/src/master/mdb/optionlists/stam...
shows the settings used to compile for the SkyLake nodes of the Stampede2 cluster at TACC using the Intel compiler.
I noticed that you are using a self-compiled MPI stack.
Usually on clusters the admins will provide an MPI stack optimized for the cluster hardware (eg to make sure Infiniband interconnects are used rather than Ethernet). I would expect better performance using those than using a self-compiled MPI stack (the Intel MPI stack will do in that respect since it is always provided by the admins and never self-compiled). Details on how to use it depend on the cluster and you would have to consult the cluster websites and / or the output of the cluster's mpicc -showme:compile and mpicc -showme:link commands (for OpenMPI mvapich and intel-mpi have similar options) to find out which libraries the official mpi compiler wrapper would use.
Yours, Roland
Dear Sir,
Please find the config-info file attached herewith...
Yours,
Vaishak
On Mon, Oct 7, 2019 at 7:56 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
hmm, still very slow.
One question that I forgot to ask before: did you make sure to build and optimized Cactus executable (setting OPTIMISE=yes, DEBUG=no to ensure that you have -O2 or -O3 optimisation settings enabled)?
Ideally if you could send the file configs/sim/config-info that would tell me.
Yours, Roland
Dear Sir,
I am a little worried about the performance because this is a new
cluster
we have and it is supposed to be performing well. I am inclined to
think
that some libraries/compiler options / settings might be the
bottleneck.
I am presently running two simulations, both using the same parameter
file
GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation
mentioned
in the previous thread. I am using one node consisting of 2*16
cores, and
32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs, distributed among 16 mpiprocs and 8 openmp threads per mpiproc.
Since I
have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these
runs.
Thanking you
Regards, Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland rhaas@illinois.edu
wrote:
Hello Vaishak,
I do not see anything obviously wrong with the setup.
It uses 128 MPI ranks for the 4 nodes which fits with there being
2x16
cores per node.
Lookin at the timer tree output at iteration 1024 (search for "gettimeof " and you will find the spot) out of 5977s spend during Evolve about 2143s were spent in "syncs" which is communication and about the same amount of time in "thorns" that is doing
computation.
While this ratio is not great (spending more time sending data than doing computation) it is also not unheard of.
Getting the original output files for the gallery data from Zenodo (link is on the gallery page):
wget
https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
you can see (in GW150914_28/output-0000/GW150914_28.out) that that
one
took about 137s for syncs and 198s for thorns, so the same ratio
but
about a factor of 10 faster.
I am reaching for straws here, but sometimes having too many MPI
ranks
can be detrimental if there is not enough work to split up (OpenMP
can
be a bit more forgiving in that respect, the original gallery run used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
Since each node has lots of RAM (more than the 96GB required to
run the
simulation), can you try and see what would happen if you were to
run
on only a single node?
Also if you could add the parameter:
Carpet::output_timers_every = 1024
then provide the files carpet-timing-statistics*.asc that would
let us
know in even more detail where the time is spent.
Running for a short time (2048 iterations) is enough to get data to compare.
Yours, Roland
Dear All,
I am running the simulation GW150914 using the parameter file
available
at
the ETK gallery at (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html) using 128
cores.
Each compute node consists of 2 X 16 Cores Intel SkyLake (
Intel(R)
Xeon(R)
Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am
running
Einstein Toolkit without OpenMP and using mpich-3.3.1.
The issue is that the simulation seems to be running at a very
slow
pace.
The number of physical time per hour that it is completing is
only
about
1.3 units. At this rate to complete 1700 units, it would take
about
54
days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630
v3 @
2.40GHz) as per the details at the example run of GW150914
available
at
the
gallery (GW150914-ETK gallery https://einsteintoolkit.org/gallery/bbh/index.html).
I have also tried using intel mpi (impi) but with simular
results.
I am also attaching the out file from the simulation.
Looking forward to your inputs.
Thanks and regards,
Vaishak P
PhD Scholar, Shyama Prasad Mukherjee Fellow Inter-University Center for Astronomy and Astrophysics (IUCAA) Pune, India
-- My email is as private as my paper mail. I therefore support
encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu
.
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Vaishak,
I have made the changes as suggested. In fact I compiled using a Intel MPI ( which was compiled by myself locally) using the optionlist of Stanpede 2 cluster as suggested by you , without OpenMP and with appropriate library paths.I am glad that the speed has improved. I am now getting around 25 physical units per hour instead of 1.5 for a simulation running on 128 mpi procs.
Glad to hear that.
The optimization parameters I am using are same as in the Stanpede2 cluster ( -Ofast -march=native) and not "-O3 -march=native" . Would it make any difference?
-Ofast can be faster than -O3 (likely it is) at the cost of reducing gcc's adherence to the IEEE floating point standard for C/C++. Basically it allows gcc to perform value unsafe optimizations (see https://gcc.gnu.org/wiki/FloatingPointMath) eg turning a+(b+c) into (a+b)+c or assuming that no infinities or NaNs will happen.
These optimizations are often fine (and the Intel compiler performs a subset of these by default) but can occasionally (see https://bitbucket.org/einsteintoolkit/tickets/issues/2119/the-binary-neutron...) lead to unexpected results.
You should be fine using them I would think. Though you may want to run at least one simulation with both options and compare results.
I have not tried these changes on mpich-3.3.1 or openmpi yet.
If Intel MPI works for you, then using it is fine.
Also where can I find more information to know about these optimization parameters?
The gcc wiki link I provided above has an discussion about what floating point optimization parameters the compiler offers. There's also https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html which goes into great detail.
If you need the last bit of performance you can also set NDEBUG=1 (which disables assert() statements) and CARPET_OPTIMIZE=1 in your CPPFLAGS which will skip some internal consistency checks in Carpet, and you can try and turn of Carpet's poisoning of uninitialized data (see the poison parameters in Carpet and CarpetLib's param.ccl files).
For options that you can set in the option list, I would try and have a look at: https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine and the references therein.
Thankyou very much for your time. This was really helpful!!
No problem.
Yours, Roland
Dear Sir,
I am surprised with the differences between results of -O3 and -Ofast!
1. In the link ( https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine), I see that not only optimization flags but also CFLAGS, CXXFLAGS, FPPFLAGS are set to ( -g3 -march=native ...) . Would it help to change these too?
2. Also, in CFLAGS and CXXFLAGS, "-std=gnu99" and "-std=gnu++0x " are used ( -std=gnu++11 in mine). Other alternatives I am aware of are "c99" and "c++11" . What are good safe choices for these keeping in mind that I am using intel libraries??
3. I am leaving CPP_OPTIMIZE_FLAGS blank. On the link (configurina a new machine), they use "-DKRANC_VECTORS". Should I consider this?
Thanking you ,
Yours Sincerely Vaishak
On Tue, Oct 8, 2019 at 11:21 PM Haas, Roland rhaas@illinois.edu wrote:
Hello Vaishak,
I have made the changes as suggested. In fact I compiled using a Intel
MPI
( which was compiled by myself locally) using the optionlist of Stanpede
2
cluster as suggested by you , without OpenMP and with appropriate library paths.I am glad that the speed has improved. I am now getting around 25 physical units per hour instead of 1.5 for a simulation running on 128
mpi
procs.
Glad to hear that.
The optimization parameters I am using are same as in the Stanpede2 cluster ( -Ofast -march=native) and not "-O3 -march=native" . Would it
make
any difference?
-Ofast can be faster than -O3 (likely it is) at the cost of reducing gcc's adherence to the IEEE floating point standard for C/C++. Basically it allows gcc to perform value unsafe optimizations (see https://gcc.gnu.org/wiki/FloatingPointMath) eg turning a+(b+c) into (a+b)+c or assuming that no infinities or NaNs will happen.
These optimizations are often fine (and the Intel compiler performs a subset of these by default) but can occasionally (see
https://bitbucket.org/einsteintoolkit/tickets/issues/2119/the-binary-neutron...) lead to unexpected results.
You should be fine using them I would think. Though you may want to run at least one simulation with both options and compare results.
I have not tried these changes on mpich-3.3.1 or openmpi yet.
If Intel MPI works for you, then using it is fine.
Also where can I find more information to know about these optimization parameters?
The gcc wiki link I provided above has an discussion about what floating point optimization parameters the compiler offers. There's also https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html which goes into great detail.
If you need the last bit of performance you can also set NDEBUG=1 (which disables assert() statements) and CARPET_OPTIMIZE=1 in your CPPFLAGS which will skip some internal consistency checks in Carpet, and you can try and turn of Carpet's poisoning of uninitialized data (see the poison parameters in Carpet and CarpetLib's param.ccl files).
For options that you can set in the option list, I would try and have a look at: https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine and the references therein.
Thankyou very much for your time. This was really helpful!!
No problem.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
Hello Vaishak,
- In the link (
https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine), I see that not only optimization flags but also CFLAGS, CXXFLAGS, FPPFLAGS are set to ( -g3 -march=native ...) . Would it help to change these too?
-g3 gives you extra debugging information. Usually just -g is enough. -march=native is important to set (if possible given your cluster setup), since otherwise the compiler will compile for the simplest x86_64 processor available (something like a core 2 from 2005) which gives sub-optimal performance.
- Also, in CFLAGS and CXXFLAGS, "-std=gnu99" and "-std=gnu++0x " are used
( -std=gnu++11 in mine). Other alternatives I am aware of are "c99" and "c++11" . What are good safe choices for these keeping in mind that I am using intel libraries??
Please use the gnu variants even with the Intel compilers. The compiler accepts them and then sets preprocessor defines (_GNU_SOURCE in particular) that make some functionality available that is often used by Einstein Toolkit codes (such as M_PI for the value of pi=3.14152...).
I have updated the wiki to request gnu++11 since Cacus now requires (at least part of) C++11.
Setting only c++11 and c99 can lead to strange errors at compile time and (occasionally) hard to debug errors when eg the strdup function (not in c09) is used without a proper prototype.
- I am leaving CPP_OPTIMIZE_FLAGS blank. On the link (configurina a new
machine), they use "-DKRANC_VECTORS". Should I consider this?
I have to admit I am not sure what that option would do. My suspicion is that it is no longer needed as vectorization use by Kranc is determined at the time the code is generated, not when it is compiled. I also cannot find the string KRANC_VECTORS anywhere in the Kranc generated code or in Kranc itself.
Yours, Roland
users@lists.einsteintoolkit.org
-
Haas, Roland -
Vaishak P