Hi Ian, Carlos,

thanks a lot for your feedback. Together with Miguel Zilhao we are planning to perform these scaling tests using the Einstein Toolkit together with the McLachlan and/or our own evolution thorns on the Cosmos cluster at the University of Cambridge. If you are interested, we would be happy to make the results available to the community and to the public, e.g., on the wiki.

We were thinking about evolving a head-on collision of two black holes to avoid contamination by the initial data construction. We would restrict the output to "carpet-timing..asc" and "AllTimers*". Would you recommend any other output to monitor the performance during these tests? Please let us know if you have any other comments.

Best wishes, Helvi & Miguel

=========================================== Dr. Helvi Witek Marie-Curie Research Fellow Dep. Fisica Quantica i Astrofisica & ICCUB Universitat de Barcelona ===========================================

On Wed, May 10, 2017 at 4:40 PM, Carlos Lousto colsma@rit.edu wrote:

Agreed, but the available machines in Xsede change every 3 years or so. It would be nice if we had a way to update/add to such "live" paper with a supplementary repo(?)

Carlos Lousto

On May 10, 2017, at 10:29 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:

On 10 May 2017, at 15:31, helvi witek hwitek@icc.ub.edu wrote:

Hi everyone,

we are going to apply for HPC time using the Lean code which is largely based on the Einstein Toolkit. Among the required technical information are scaling tests. While we will perform our own tests I also checked for "official" information for the ET. I noticed that there is very little public information, e.g. on the wiki, about recent (say, within the last five years) scaling tests aside from Eloisa's recent paper http://inspirehep.net/record/1492289 and this tracker http://lists.einsteintoolkit.org/pipermail/users/2013-February/002815.html

Did I miss anything? It might be a good idea to add a standardized test or references to the wiki page.

It would probably be very useful for many groups if we were to write a short paper describing the scaling of the ET in various cases on current HPC machines. For someone running simulations very similar to those used, referring to such a paper may be sufficient to demonstrate scaling of the code for a proposal. If the code used was quite different, then if the parameter files and any required scripts from such a paper were made public, it would be easier for each group to adapt them to their own code.

-- Ian Hinder http://members.aei.mpg.de/ianhin

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Helvi, Miguel

I find that many (most)? users of the Einstein Toolkit define their problem size based on physics and accuracy requirements, and then adapt the number of nodes they uses to make this problem run most efficiently. This usually requires a certain (fixed) amount of work per core. Correspondingly, an informative benchmarking chart is a "weak scaling test". Strong scalability is also interesting, but is more difficult to interpret in the presence of adaptive mesh refinement with a complex system of equations since using too few nodes leads to out-of-memory situations, and using too many nodes quickly leads to inefficiencies if you use higher-order derivatives.

Efficiency depends also on the number of OpenMP threads vs. the number of MPI processes, and (obviously) the amount of I/O you're doing, which is governed by very different characteristics (the file system used, number of file servers, etc.) We thus typically benchmark weak scalability on setups that perform very little I/O, and where we exclude initial data generation.

When setting up a benchmark, it is important to monitor as many performance characteristics as possible to ensure that one isn't limited by I/O, or by any other "accidental" feature that could easily be removed in a production simulation such as horizon finding, constraint evaluation, too frequent regridding, etc.

It is surprisingly easy to accidentally enable a feature in the Einstein Toolkit that adversely affects performance, in particular when running on 1000+ cores. Correspondingly, one has to take great care when defining a benchmark that no such features are present. Timer output will be valuable here to understand how much

A QC-0 setup should be a good test case if you want to study BBH scenarios. Or maybe GW150914 http://einsteintoolkit.org/gallery/bbh/index.html would be more interesting and relevant? You would obviously crank up the resolution to turn this into a weak scaling test. In my experience, running for a few minutes (less than ten) after initial data setup will suffice to get good numbers.

The benchmarks I am running are usually simpler since I tend to focus on a particular features that I want to optimize (e.g. RHS evaluation, grid structure management). I think it's been some time since we ran production-scale BBH benchmarks (with puncture tracking, regridding tracking the black holes, etc.), I believe we focussed on hydrodynamics benchmarks recently.

The final quantity against which I report performance is usually "grid point updates per second", plotted against the number of nodes used. This is the time required to evaluate the RHS for a single grid point, amortized over and including all overhead such as scheduling, regridding, prolongation, synchronization, etc. (Since this measures time, smaller numbers are better.) On a fast machine, and if the overhead is low, this number can be as low as a few microseconds. Mesh refinement and parallelization will add a non-negligible overhead to this, and a weak scaling test will show when the parallelization overhead becomes prohibitive.

-erik

On Fri, May 12, 2017 at 11:41 AM, helvi witek hwitek@icc.ub.edu wrote:

Hi Ian, Carlos,

thanks a lot for your feedback. Together with Miguel Zilhao we are planning to perform these scaling tests using the Einstein Toolkit together with the McLachlan and/or our own evolution thorns on the Cosmos cluster at the University of Cambridge. If you are interested, we would be happy to make the results available to the community and to the public, e.g., on the wiki.

We were thinking about evolving a head-on collision of two black holes to avoid contamination by the initial data construction. We would restrict the output to "carpet-timing..asc" and "AllTimers*". Would you recommend any other output to monitor the performance during these tests? Please let us know if you have any other comments.

Best wishes, Helvi & Miguel

=========================================== Dr. Helvi Witek Marie-Curie Research Fellow Dep. Fisica Quantica i Astrofisica & ICCUB Universitat de Barcelona ===========================================

On Wed, May 10, 2017 at 4:40 PM, Carlos Lousto colsma@rit.edu wrote:

...

Agreed, but the available machines in Xsede change every 3 years or so. It would be nice if we had a way to update/add to such "live" paper with a supplementary repo(?)

Carlos Lousto

On May 10, 2017, at 10:29 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:

On 10 May 2017, at 15:31, helvi witek hwitek@icc.ub.edu wrote:

Hi everyone,

we are going to apply for HPC time using the Lean code which is largely based on the Einstein Toolkit. Among the required technical information are scaling tests. While we will perform our own tests I also checked for "official" information for the ET. I noticed that there is very little public information, e.g. on the wiki, about recent (say, within the last five years) scaling tests aside from Eloisa's recent paper http://inspirehep.net/record/1492289 and this tracker http://lists.einsteintoolkit.org/pipermail/users/2013-Februa ry/002815.html

Did I miss anything? It might be a good idea to add a standardized test or references to the wiki page.

It would probably be very useful for many groups if we were to write a short paper describing the scaling of the ET in various cases on current HPC machines. For someone running simulations very similar to those used, referring to such a paper may be sufficient to demonstrate scaling of the code for a proposal. If the code used was quite different, then if the parameter files and any required scripts from such a paper were made public, it would be easier for each group to adapt them to their own code.

-- Ian Hinder http://members.aei.mpg.de/ianhin

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users