Respected Sir,

I succeeded in running ETK on the machine. I conducted a few experiments and I found the following:

1. Intrinsic vectorization has to be disabled. gcc's auto vectorization with avx2 enabled then leads to successful evolution (at slower speeds).

2. Turning on intrinsic vectorization + disabling avx in gcc works for a short period of time. The evolution then stops due to punctures going to inf.

3. Intrinsic vectorization + gcc avx does not work due Seg faults at the testing vectorization stage.

I have read the page at

https://docs.einsteintoolkit.org/et-docs/Vectorisation

Is this outdated? Does intrinsic vectorization add the capabilities to use 256bit wide data types on avx2 capable machines?

Thanks and regards

On Thu, Mar 2, 2023 at 11:24 PM Roland Haas <rhaas@illinois.edu> wrote:

Hello Vaishak,

Sorry for the delay, and thank you for including the various log files.

I have been running on a new AMD based system (NCSA Delta, Milan, not
Rome) during the last week (with Vectors active), though it is a
slightly older ET code (no changes to Vectors though). I also ran on
SDSC Expanse (Rome, Epyc 7742) for the ET testsuite for the 2022_11
release (http://einsteintoolkit.org/testsuite_results/index.php)
without SEGFAULT failures.

This unfortunately makes debugging the issue that you are facing harder.

One (possible) issue could be related to using -march=native in you
compilation flags. Since this instructs GCC to compile for the CPU
architecture it finds itself running on, I would double check that
indeed the login nodes on sonic use the same CPU as the compute nodes.

Yours,
Roland

> Dear All,
>
> Greetings from India. I am trying to get the ETK working on an AMD Rome
> powered supercomputer at ICTS, India. I am working with gcc (11.1.0,
> 12.2.0) and openmpi. The compilation is successful but every one of the
> tests and runs fails due to seg faults at the vectorization stage. On
> recompiling the toolkit without vectorization, the tests run (except for
> one test of ML_BSSN which fails due to a relative error ~ 1e-14). I am
> attaching the backtrace (from the gallery BBH run), make log (with
> vectorization) and the optionlist herewith.
>
> Requesting help!
>
>
> With regards
>

--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://keys.gnupg.net.