[Users] GPU support for the Einstein Toolkit

[Erik Schnetter]

Bruno

I hear from the AMReX developers that others have run AMReX-based
codes on the GPUs on Marconi. So all is good.

-erik

On Fri, Jul 10, 2020 at 11:56 AM Bruno Giacomazzo
<bruno.giacomazzo at unimib.it> wrote:
>
> Erik,
>   thanks a lot for the information and for the work you are doing. I will contact CINECA and ask about AMReX support.
>
> Cheers,
> Bruno
>
>
> Il giorno ven 10 lug 2020 alle ore 17:53 Erik Schnetter <schnetter at cct.lsu.edu> ha scritto:
>>
>> Bruno
>>
>> We are looking into GPU support in the context of the CarpetX project.
>> CarpetX <https://bitbucket.org/eschnett/cactusamrex/src/master/> is a
>> new driver, currently being developed, that uses AMReX
>> <https://amrex-codes.github.io/amrex/> as back-end to provide the
>> lower-level AMR capabilities. AMReX supports GPUs, and we intend to
>> ensure that CarpetX (and thus the Einstein Toolkit) supports GPUs from
>> the beginning. I am (personally) currently not using the larger EU HPC
>> systems, but we have Summit <https://www.olcf.ornl.gov/summit/> in the
>> North American region, and Summit requires highly efficient GPU codes
>> as well.
>>
>> Having said this, CarpetX is still in the very early stages, so I can
>> only present a plan and not yet a solution. We know that AMReX
>> supports GPUs, and AMReX is funded and actively developed, so we have
>> a very well engineered foundation, but exactly how to expose this to
>> Cactus and the Einstein Toolkit isn't quite clear yet. My hope
>> (dream?) is that it will be possible to write somewhat generic C++
>> code that will run unchanged on both CPUs and GPUs, and that making
>> this work efficiently will work without having to maintain two
>> different codes. Whether this is possible will remain to be seen.
>> Maybe we need to go towards automated code generation (generating both
>> CPU and GPU codes from the same script) to make this happen.
>>
>> I don't see AMReX on the list of supported software for Marconi. Do
>> you want to investigate? I wonder whether this is due to lack of
>> interest from users, or whether they looked into AMReX and there was a
>> roadblock. (I would not assume so, but you never know.)
>>
>> -erik
>>
>> On Fri, Jul 10, 2020 at 11:11 AM Bruno Giacomazzo
>> <bruno.giacomazzo at unimib.it> wrote:
>> >
>> > Dear All,
>> >   I would like to know if there is any plan to port the Einstein Toolkit to GPUs. This is something that we may also discuss at the next Einstein Toolkit workshop if there is time.
>> >
>> >   The reason I'm asking is that in Italy, CINECA (part of the EU HPC consortium) just bought its new HPC cluster and it is completely GPU based (Marconi100, https://www.hpc.cineca.it/hardware/marconi100, currently ranked in the top 10 of the fastest supercomputers in the world).
>> >
>> >    Speaking with people at CINECA, I was informed that HPC-Europe is moving to GPU and, while cpu codes will be still supported for the next 5 years (with some of the new clusters still having a limited number of nodes for cpu codes), the plan is to move to 100% GPU clusters at the EU level. I was indeed told that the latest 3 HPC clusters bought by HPC-Europe are GPU clusters. The next Italian HPC machine will also be a GPU cluster, with a limited number of cpu nodes (they estimate ~3500 nodes with GPUs and ~1000 nodes with cpus).
>> >
>> > Thanks,
>> > Bruno
>> >
>> > --
>> >
>> > Prof. Bruno Giacomazzo
>> > Department of Physics
>> > University of Milano-Bicocca
>> > Piazza della Scienza 3
>> > 20126 Milano
>> > Italy
>> >
>> > email: bruno.giacomazzo at unimib.it
>> > phone: (+39) 02 6448 2321
>> > web: http://www.brunogiacomazzo.org
>> >
>> > ----------------------------------------------------------------------
>> > There are only 10 types of people in the world:
>> > Those who understand binary, and those who don't
>> > ----------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users at einsteintoolkit.org
>> > http://lists.einsteintoolkit.org/mailman/listinfo/users
>>
>>
>>
>> --
>> Erik Schnetter <schnetter at cct.lsu.edu>
>> http://www.perimeterinstitute.ca/personal/eschnetter/
>
>
>
> --
>
> Prof. Bruno Giacomazzo
> Department of Physics
> University of Milano-Bicocca
> Piazza della Scienza 3
> 20126 Milano
> Italy
>
> email: bruno.giacomazzo at unimib.it
> phone: (+39) 02 6448 2321
> web: http://www.brunogiacomazzo.org
>
> ----------------------------------------------------------------------
> There are only 10 types of people in the world:
> Those who understand binary, and those who don't
> ----------------------------------------------------------------------



-- 
Erik Schnetter <schnetter at cct.lsu.edu>
http://www.perimeterinstitute.ca/personal/eschnetter/