<div dir="ltr">Bruno<div><br></div><div>The amount of data output varies greatly between simulations, as well as the intervals between output. And usually, I/O scales very differently than the computation itself. Therefore, it is customary (at least while trying to understand results) to test evolution and output separately. That is, the evolution benchmark would probably output the maximum of rho only, and an I/O benchmark would output (and/or recover) just Minkowski data without time evolution.</div>
<div><br></div><div style>Regarding the number of threads: To determine the ideal number of threads to use, we run single-node benchmarks of code that is well parallellised via OpenMP. Code that is not really parallel (e.g. some initial data routines) would distort these results, as would using a large number of cores, since this involves also parallel scalability. Using this "ideal" number of cores, we then run scalability benchmarks to see where parallel scaling breaks down.</div>
<div style><br></div><div style>For any given physics situation, one then has to strike a compromise:</div><div style>- non-parallel sections of the code prefer using 1 OpenMP thread</div><div style>- parallel MPI scaling prefers using as few MPI processes as possible, i.e. using many OpenMP threads</div>
<div style>The balance thus shifts with the number of cores -- the more cores you use, the more OpenMP threads you will also want to use to counter-act MPI scaling problems.</div><div style><br></div><div style>As others have said, binding threads to cores and binding memory to processes is also very important, and is visible in particular on single-node benchmarks.</div>
<div style><br></div><div style>For any benchmark I run, I also look at detailed timer output. The standard Cactus timers are not good enough for this; you will have to use TimerReport or Carpet's timers for this. One interesting quantity is to see what fraction of the time is spent in the actual evolution thorns (not just CCTK_EVOL; this also measures some infrastructure tasks). The other interesting effect to watch is how this time distribution changes as the number of MPI processes increases -- this shows the effect of scaling problems. The latter should e.g. show that ASCII output becomes more time consuming, or that synchronisation or load balancing takes more time.</div>
<div style><br></div><div style>Finally, I found it very difficult to come up with a "good" benchmark parameter file. Such a benchmark should run both on few and on many cores, should contains all the relevant thorns, should not do I/O, should not contain anything that is known not to scale, should not encounter nans or con2prim problems, should be "close" to actual parameter files that people actually want to use, etc.</div>
<div><br></div><div style>I think it's time to create a wiki page for benchmarking! There we could describe (a) the tools available (timers, etc.), (b) the pitfalls to avoid (e.g. measure evolution and I/O separately), and (c) discuss results that we find.</div>
<div style><br></div><div style>In this case -- I think this means we should improve ASCII output! Writing ASCII files is always slow, but collecting data onto a single process shouldn't be. That's no more than a reduction operation, and with InfiniBand bandwidths of tens of Gigabytes per second, we are very far away from what the hardware performance allows us to do. Let's open a bug report for this. We should either correct this, or should people prominently warn about this.</div>
<div><br></div><div>-erik</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 21, 2013 at 8:31 PM, Bruno Giacomazzo <span dir="ltr"><<a href="mailto:bruno.giacomazzo@jila.colorado.edu" target="_blank">bruno.giacomazzo@jila.colorado.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
I did some strong scaling tests on Stampede (<a href="https://www.xsede.org/tacc-stampede" target="_blank">https://www.xsede.org/tacc-stampede</a>) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs.<br>
<br>
I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs).<br>
<br>
First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it?<br>
<br>
According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede?<br>
<br>
Let me know if you have any comment or suggestion.<br>
<br>
Cheers,<br>
Bruno<br>
<br>
Dr. Bruno Giacomazzo<br>
JILA - University of Colorado<br>
440 UCB<br>
Boulder, CO 80309<br>
USA<br>
<br>
Tel. : +1 303-492-5170<br>
Fax : +1 303-492-5235<br>
email : <a href="mailto:bruno.giacomazzo@jila.colorado.edu">bruno.giacomazzo@jila.colorado.edu</a><br>
web: <a href="http://www.brunogiacomazzo.org" target="_blank">http://www.brunogiacomazzo.org</a><br>
<br>
----------------------------------------------------------------------<br>
There are only 10 types of people in the world:<br>
Those who understand binary, and those who don't<br>
----------------------------------------------------------------------<br>
<br>
<br><br>
<br><br>
<br><br>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>