Bruno
I hear from the AMReX developers that others have run AMReX-based codes on the GPUs on Marconi. So all is good.
-erik
On Fri, Jul 10, 2020 at 11:56 AM Bruno Giacomazzo bruno.giacomazzo@unimib.it wrote:
Erik, thanks a lot for the information and for the work you are doing. I will contact CINECA and ask about AMReX support.
Cheers, Bruno
Il giorno ven 10 lug 2020 alle ore 17:53 Erik Schnetter schnetter@cct.lsu.edu ha scritto:
Bruno
We are looking into GPU support in the context of the CarpetX project. CarpetX https://bitbucket.org/eschnett/cactusamrex/src/master/ is a new driver, currently being developed, that uses AMReX https://amrex-codes.github.io/amrex/ as back-end to provide the lower-level AMR capabilities. AMReX supports GPUs, and we intend to ensure that CarpetX (and thus the Einstein Toolkit) supports GPUs from the beginning. I am (personally) currently not using the larger EU HPC systems, but we have Summit https://www.olcf.ornl.gov/summit/ in the North American region, and Summit requires highly efficient GPU codes as well.
Having said this, CarpetX is still in the very early stages, so I can only present a plan and not yet a solution. We know that AMReX supports GPUs, and AMReX is funded and actively developed, so we have a very well engineered foundation, but exactly how to expose this to Cactus and the Einstein Toolkit isn't quite clear yet. My hope (dream?) is that it will be possible to write somewhat generic C++ code that will run unchanged on both CPUs and GPUs, and that making this work efficiently will work without having to maintain two different codes. Whether this is possible will remain to be seen. Maybe we need to go towards automated code generation (generating both CPU and GPU codes from the same script) to make this happen.
I don't see AMReX on the list of supported software for Marconi. Do you want to investigate? I wonder whether this is due to lack of interest from users, or whether they looked into AMReX and there was a roadblock. (I would not assume so, but you never know.)
-erik
On Fri, Jul 10, 2020 at 11:11 AM Bruno Giacomazzo bruno.giacomazzo@unimib.it wrote:
Dear All, I would like to know if there is any plan to port the Einstein Toolkit to GPUs. This is something that we may also discuss at the next Einstein Toolkit workshop if there is time.
The reason I'm asking is that in Italy, CINECA (part of the EU HPC consortium) just bought its new HPC cluster and it is completely GPU based (Marconi100, https://www.hpc.cineca.it/hardware/marconi100, currently ranked in the top 10 of the fastest supercomputers in the world).
Speaking with people at CINECA, I was informed that HPC-Europe is moving to GPU and, while cpu codes will be still supported for the next 5 years (with some of the new clusters still having a limited number of nodes for cpu codes), the plan is to move to 100% GPU clusters at the EU level. I was indeed told that the latest 3 HPC clusters bought by HPC-Europe are GPU clusters. The next Italian HPC machine will also be a GPU cluster, with a limited number of cpu nodes (they estimate ~3500 nodes with GPUs and ~1000 nodes with cpus).
Thanks, Bruno
--
Prof. Bruno Giacomazzo Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy
email: bruno.giacomazzo@unimib.it phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.org
There are only 10 types of people in the world: Those who understand binary, and those who don't
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
--
Prof. Bruno Giacomazzo Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy
email: bruno.giacomazzo@unimib.it phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.org
There are only 10 types of people in the world: Those who understand binary, and those who don't