[Users] Setting up ETK on an AMD Rome cluster

[Vaishak Prasad]

Respected Sir,

I succeeded in running ETK on the machine. I conducted a few experiments
and I found the following:

1. Intrinsic vectorization has to be disabled. gcc's auto vectorization
with avx2 enabled then leads to successful evolution (at slower speeds).
2. Turning on intrinsic vectorization + disabling avx in gcc works for a
short period of time. The evolution then stops due to punctures going to
inf.
3. Intrinsic vectorization + gcc avx does not work due Seg faults at the
testing vectorization stage.

I have read the page at

https://docs.einsteintoolkit.org/et-docs/Vectorisation

Is this outdated? Does intrinsic vectorization add the capabilities to use
256bit wide data types on avx2 capable machines?


Thanks and regards

On Thu, Mar 2, 2023 at 11:24 PM Roland Haas <rhaas at illinois.edu> wrote:

> Hello Vaishak,
>
> Sorry for the delay, and thank you for including the various log files.
>
> I have been running on a new AMD based system (NCSA Delta, Milan, not
> Rome) during the last week (with Vectors active), though it is a
> slightly older ET code (no changes to Vectors though). I also ran on
> SDSC Expanse (Rome, Epyc 7742) for the ET testsuite for the 2022_11
> release (http://einsteintoolkit.org/testsuite_results/index.php)
> without SEGFAULT failures.
>
> This unfortunately makes debugging the issue that you are facing harder.
>
> One (possible) issue could be related to using  -march=native in you
> compilation flags. Since this instructs GCC to compile for the CPU
> architecture it finds itself running on, I would double check that
> indeed the login nodes on sonic use the same CPU as the compute nodes.
>
> Yours,
> Roland
>
> > Dear All,
> >
> > Greetings from India. I am trying to get the ETK working on an AMD Rome
> > powered supercomputer at ICTS, India. I am working with gcc (11.1.0,
> > 12.2.0) and openmpi. The compilation is successful but every one of the
> > tests and runs fails due to seg faults at the vectorization stage. On
> > recompiling the toolkit without vectorization, the tests run  (except for
> > one test of ML_BSSN which fails due to a relative error ~ 1e-14). I am
> > attaching the backtrace (from the gallery BBH run), make log (with
> > vectorization) and the optionlist herewith.
> >
> > Requesting help!
> >
> >
> > With regards
> >
>
>
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://keys.gnupg.net.
>


-- 
Vaishak Prasad

PhD Scholar and
Shyama Prasad Mukherjee Fellow
Inter-University Center for Astronomy and Astrophysics (IUCAA)
Pune, India
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20230302/aba2c764/attachment.html