<div dir="ltr"><div>Respected Sir,</div><div><br></div><div>I succeeded in running ETK on the machine. I conducted a few experiments and I found the following:<br></div><div><br></div><div>1. Intrinsic vectorization has to be disabled. gcc's auto vectorization with avx2 enabled then leads to successful evolution (at slower speeds).</div><div>2. Turning on intrinsic vectorization + disabling avx in gcc works for a short period of time. The evolution then stops due to punctures going to inf.</div><div>3. Intrinsic vectorization + gcc avx does not work due Seg faults at the testing vectorization stage. <br></div><div><br></div><div>I have read the page at <br></div><div><br></div><div><a href="https://docs.einsteintoolkit.org/et-docs/Vectorisation" target="_blank">https://docs.einsteintoolkit.org/et-docs/Vectorisation</a></div><div><br></div><div>Is this outdated? Does intrinsic vectorization add the capabilities to use 256bit wide data types on avx2 capable machines?</div><div><br></div><div><br></div><div>Thanks and regards<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023 at 11:24 PM Roland Haas <<a href="mailto:rhaas@illinois.edu" target="_blank">rhaas@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Vaishak,<br>
<br>
Sorry for the delay, and thank you for including the various log files.<br>
<br>
I have been running on a new AMD based system (NCSA Delta, Milan, not<br>
Rome) during the last week (with Vectors active), though it is a<br>
slightly older ET code (no changes to Vectors though). I also ran on<br>
SDSC Expanse (Rome, Epyc 7742) for the ET testsuite for the 2022_11<br>
release (<a href="http://einsteintoolkit.org/testsuite_results/index.php" rel="noreferrer" target="_blank">http://einsteintoolkit.org/testsuite_results/index.php</a>)<br>
without SEGFAULT failures.<br>
<br>
This unfortunately makes debugging the issue that you are facing harder.<br>
<br>
One (possible) issue could be related to using -march=native in you<br>
compilation flags. Since this instructs GCC to compile for the CPU<br>
architecture it finds itself running on, I would double check that<br>
indeed the login nodes on sonic use the same CPU as the compute nodes.<br>
<br>
Yours,<br>
Roland<br>
<br>
> Dear All,<br>
> <br>
> Greetings from India. I am trying to get the ETK working on an AMD Rome<br>
> powered supercomputer at ICTS, India. I am working with gcc (11.1.0,<br>
> 12.2.0) and openmpi. The compilation is successful but every one of the<br>
> tests and runs fails due to seg faults at the vectorization stage. On<br>
> recompiling the toolkit without vectorization, the tests run (except for<br>
> one test of ML_BSSN which fails due to a relative error ~ 1e-14). I am<br>
> attaching the backtrace (from the gallery BBH run), make log (with<br>
> vectorization) and the optionlist herewith.<br>
> <br>
> Requesting help!<br>
> <br>
> <br>
> With regards<br>
> <br>
<br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://keys.gnupg.net" rel="noreferrer" target="_blank">http://keys.gnupg.net</a>.<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Vaishak Prasad<br><br>PhD Scholar and <br></div><div dir="ltr">Shyama Prasad Mukherjee Fellow<br>
Inter-University Center for Astronomy and Astrophysics (IUCAA)<br>
Pune, India<div><br></div></div></div></div></div>