<div dir="ltr">Hi everyone,<div><br></div><div>Thanks Jim for your message. </div><div><br></div><div>I ran some new tests on 2 and 3 nodes. </div><div><br></div><div>* The only way I can get some form of scaling from 2 to 3 is</div><div> with OpenMPI 3.1.6</div><div>* With 2 nodes, the Intel compiler is faster.</div><div>* With 3 nodes, there is no difference in speed between Intel + </div><div> OpenMPI 3.1.6 and GNU 9 + OpenMPI 3.1.6. </div><div>* On 2 nodes, using mvapich is faster than using OpenMPI<br></div><div>* On 2 nodes, GNU 10 + OpenMPI 4.0.4 is as fast as </div><div> GNU 9 + OpenMPI 3.1.6.</div><div><br></div><div>It looks like that internode connection works best with </div><div>OpenMPI 3.1.6. </div><div><br></div><div>> 1. Your tests of Intel19 combinations include hydro simulations? I've only tested for binary NS mergers and got NaNs from the beginning. If you have succeeded in hydro simulations, would you let me know the runtime options?<br></div><div><br></div><div>I successfully ran static_tov on Expanse with the intel stack.</div><div>I don't think I used any particular option.</div><div><br></div><div><div>> 2. In those comibations, I think there would be no avx options available for the AMD cpus. Can I get the configuration file for the AMD+Intel19?</div><div><br></div><div>The intel compiler supports AVX also on non-intel CPUs.</div><div>You can enable that with -march=core-avx2.</div><div><br></div></div><div>Gabriele</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 27, 2021 at 4:47 PM Hee Il Kim <<a href="mailto:heeilkim@gmail.com">heeilkim@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hello Jim,</div><div dir="ltr"><br></div><div>Recently I posted a mail on Intel compiler issues. </div><div dir="ltr"><div><a href="http://lists.einsteintoolkit.org/pipermail/users/2021-August/008175.html" target="_blank">http://lists.einsteintoolkit.org/pipermail/users/2021-August/008175.html</a></div><div><br></div><div>I have two questions.</div><div><br></div><div>1. Your tests of Intel19 combinations include hydro simulations? I've only tested for binary NS mergers and got NaNs from the beginning. If you have succeeded in hydro simulations, would you let me know the runtime options?</div><div><br></div><div>2. In those comibations, I think there would be no avx options available for the AMD cpus. Can I get the configuration file for the AMD+Intel19?</div><div><br></div><div>Thanks you.</div><div><br></div><div>Hee Il</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 28, 2021 at 7:07 AM James Healy <<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hello all,</div>
<div><br>
</div>
<div>TLDR: Try the combination intel/<a href="http://19.1.1.217" target="_blank">19.1.1.217</a>
and openmpi/3.1.6 on expanse.<br>
</div>
<div><br>
</div>
<div>Long version:</div>
<div><br>
</div>
<div>I had a similar issue on expanse during
the EUP with great performance on 1 node and worse on multiple
nodes. I tried many combinations of the available (at the time)
mpi implementations and compilers. <br>
</div>
<div><br>
</div>
<div>Here is what I found:</div>
<div>
<pre>* intel19 + intel mpi : great 1 node speed, poor scaling to >1 node, poor weak scaling
* intel19 + openmpi 4 : same as above
* gcc 10 + openmpi 4 : 40% slower 1 node speed, poor scaling to >1 node, poor weak scaling
* gcc 9 + openmpi 3.1.6 : 40% slower 1 node speed, good scaling to >1 node, acceptable weak scaling
* using intel19 + openmpi 4 executable but with gcc9/openmpi 3.1.6 modules loaded at runtime : great 1 node speed, good scaling to >1 node, acceptable weak scaling</pre>
</div>
<div>The MPI implementation that worked was
using openmpi 3.1.6. At the time this was only available if you
used the gcc 9 compilers. However, I contacted support and they
installed a version for the intel compilers. Here is what support
said:</div>
<div>
<pre>"I think I can install openmpi/3.1.6 with intel compilers. I have to go back and check but I think the main difference is we are using ibverbs on the openmpi/3.1.6 build and ucx on the openmpi/<a href="http://4.0.4." target="_blank">4.0.4.</a> For most codes ucx has been the faster option but in your case it seems different. I will let you know once the compilers are in place."
</pre>
</div>
<div>After he installed it, the combination
of intel19 compilers with openmpi 3.1.6 gives acceptable scaling.
I am not familiar enough with ucx vs ibverbs to comment on if that
is the issue with the AMD clusters. I also have this same issue
on bridges2 which uses the same AMD nodes as expanse, and have not
been able to get the code to perform well on >1 node.</div>
<div><br>
</div>
<div>At least for expanse, I'd suggest
trying to load intel 19 and openmpi and see if you get better
scaling. Then if you do, we could inquire with support on the
differences in configurations for openmpi 3.1.6 and openmpi 4.0.4
to see if there is more beyond ucx vs ibverbs. Then, the next
step would be seeing if we can replicate this elsewhere (like
bridges2 or anvil).<br>
</div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">module load
intel/<a href="http://19.1.1.217" target="_blank">19.1.1.217</a><br>
module load openmpi/3.1.6</font></div>
<br>
<div>Best,<br>
Jim Healy</div>
<div>CCRG Research Associate<br>
</div>
<div><br>
</div>
<div>On 8/27/21 12:44 PM, Gabriele Bozzola
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello,
<div><br>
</div>
<div>Last week I opened a PR to add the configuration files </div>
<div>for Expanse to simfactory. Expanse is an example of</div>
<div>the new generation of AMD supercomputers. Others are</div>
<div>Anvil, one of the other new XSEDE machines, or Puma, </div>
<div>the newest cluster at The University of Arizona.</div>
<div><br>
</div>
<div>I have some experience with Puma and Expanse and</div>
<div>I would like to share some thoughts, some of which come</div>
<div>from interacting with the admins of Expanse. The problem</div>
<div>is that I am finding terrible multi-node performance on
both </div>
<div>these machines, and I don't know if this will be a common</div>
<div>thread among new AMD clusters. </div>
<div><br>
</div>
<div>These supercomputers have similar characteristics.<br>
</div>
<div><br>
</div>
<div>First, they have very high cores/node count (typically </div>
<div>128/node) but low memory per core (typically 2 GB / core).</div>
<div>In these conditions, it is very easy to have a job killed
by </div>
<div>the OOM daemon. My suspicion is that it is rank 0 that </div>
<div>goes out of memory, and the entire run is aborted.</div>
<div><br>
</div>
<div>Second, depending on the MPI implementation, MPI collective</div>
<div>operations can be extremely expensive. I was told that</div>
<div>the best implementation is mvapich 2.3.6 (at the moment).</div>
<div>This seems to be due to the high core count.</div>
<div><br>
</div>
<div>I found that the code does not scale well. This is
possibly </div>
<div>related to the previous point. If your job can fit on a
single node, </div>
<div>it will run wonderfully. However, when you perform the
same </div>
<div>simulation on two nodes, the code will actually be slower. </div>
<div>This indicates that there's no strong scaling at all from </div>
<div>1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks).</div>
<div>Using mvapich 2.3.6 improves the situation, but it is still</div>
<div>faster to use fewer nodes. </div>
<div><br>
</div>
<div>(My benchmark is a par file I've tested extensively on
Frontera)<br>
</div>
<div><br>
</div>
<div>I am working with Expanse's support staff to see what we
can</div>
<div>do, but I wonder if anyone has had a positive experience
with </div>
<div>this architecture and has some tips to share.</div>
<div>
<div><br>
</div>
</div>
<div>Gabriele</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<p><br>
</p>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote></div>
</blockquote></div>