[Users] strong scaling tests on Stampede (TACC)

[Bruno Giacomazzo]

Hi,
	I did some strong scaling tests on Stampede (https://www.xsede.org/tacc-stampede) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs.

	I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs).

	First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it?

	According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede?

	Let me know if you have any comment or suggestion.

Cheers,
Bruno

Dr. Bruno Giacomazzo
JILA - University of Colorado
440 UCB
Boulder, CO 80309
USA

Tel.  : +1 303-492-5170
Fax  : +1 303-492-5235
email : bruno.giacomazzo at jila.colorado.edu
web: http://www.brunogiacomazzo.org

----------------------------------------------------------------------
There are only 10 types of people in the world:
Those who understand binary, and those who don't
----------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: GRH_RL4_dx1.0_Octant.par
Type: application/octet-stream
Size: 8666 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20130221/15b5ffd5/attachment-0001.obj 
-------------- next part --------------

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: TimingStampede_Dev.txt
Url: http://lists.einsteintoolkit.org/pipermail/users/attachments/20130221/15b5ffd5/attachment-0001.txt 
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot_scaling_walltime_all.pdf
Type: application/pdf
Size: 5716 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20130221/15b5ffd5/attachment-0002.pdf 
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot_scaling_cctkevol.pdf
Type: application/pdf
Size: 5439 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20130221/15b5ffd5/attachment-0003.pdf