<div dir="ltr"><div>Dear Sir,</div><div><br></div><div>I am surprised with the differences between results of -O3 and -Ofast!<br></div><div><br></div><div>1. In the link (<a href="https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine" rel="noreferrer" target="_blank">https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine</a>), I see that not only optimization flags but also CFLAGS, CXXFLAGS, FPPFLAGS are set to ( -g3 -march=native ...) . Would it help to change these too? <br></div><div><br></div><div>2. Also, in CFLAGS and CXXFLAGS, "-std=gnu99" and "-std=gnu++0x " are used ( -std=gnu++11 in mine). Other alternatives I am aware of are "c99" and "c++11" . What are good safe choices for these keeping in mind that I am using intel libraries??</div><div><br></div><div>3. I am leaving CPP_OPTIMIZE_FLAGS blank. On the link (configurina a new machine), they use "-DKRANC_VECTORS". Should I consider this? </div><div><br></div><div><br></div><div><br></div><div>Thanking you ,</div><div><br></div><div>Yours Sincerely</div><div>Vaishak<br></div><div> <br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 8, 2019 at 11:21 PM Haas, Roland <<a href="mailto:rhaas@illinois.edu">rhaas@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Vaishak,<br>
<br>
> I have made the changes as suggested. In fact I compiled using a Intel MPI<br>
> ( which was compiled by myself locally) using the optionlist of Stanpede 2<br>
> cluster as suggested by you , without OpenMP and with appropriate library<br>
> paths.I am glad that the speed has improved. I am now getting around 25<br>
> physical units per hour instead of 1.5 for a simulation running on 128 mpi<br>
> procs. <br>
Glad to hear that.<br>
<br>
> The optimization parameters I am using are same as in the Stanpede2<br>
> cluster ( -Ofast -march=native) and not "-O3 -march=native" . Would it make<br>
> any difference?<br>
-Ofast can be faster than -O3 (likely it is) at the cost of reducing<br>
gcc's adherence to the IEEE floating point standard for C/C++.<br>
Basically it allows gcc to perform value unsafe optimizations (see<br>
<a href="https://gcc.gnu.org/wiki/FloatingPointMath" rel="noreferrer" target="_blank">https://gcc.gnu.org/wiki/FloatingPointMath</a>) eg turning a+(b+c) into<br>
(a+b)+c or assuming that no infinities or NaNs will happen.<br>
<br>
These optimizations are often fine (and the Intel compiler performs a<br>
subset of these by default) but can occasionally (see<br>
<a href="https://bitbucket.org/einsteintoolkit/tickets/issues/2119/the-binary-neutron-star-gallery-example#comment-54050358" rel="noreferrer" target="_blank">https://bitbucket.org/einsteintoolkit/tickets/issues/2119/the-binary-neutron-star-gallery-example#comment-54050358</a>) lead to unexpected results.<br>
<br>
You should be fine using them I would think. Though you may want to run<br>
at least one simulation with both options and compare results.<br>
<br>
> I have not tried these changes on mpich-3.3.1 or openmpi yet.<br>
If Intel MPI works for you, then using it is fine.<br>
<br>
> Also where can I find more information to know about these optimization<br>
> parameters?<br>
The gcc wiki link I provided above has an discussion about what floating<br>
point optimization parameters the compiler offers. There's also<br>
<a href="https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html" rel="noreferrer" target="_blank">https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html</a> which goes<br>
into great detail.<br>
<br>
If you need the last bit of performance you can also set NDEBUG=1<br>
(which disables assert() statements) and CARPET_OPTIMIZE=1 in your<br>
CPPFLAGS which will skip some internal consistency checks in Carpet,<br>
and you can try and turn of Carpet's poisoning of uninitialized data<br>
(see the poison parameters in Carpet and CarpetLib's param.ccl files).<br>
<br>
For options that you can set in the option list, I would try and have a<br>
look at:<br>
<a href="https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine" rel="noreferrer" target="_blank">https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine</a> and the references therein.<br>
<br>
> Thankyou very much for your time. This was really helpful!!<br>
No problem.<br>
<br>
Yours,<br>
Roland<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Regards,<br>Vaishak P<br><br>PhD Scholar,<br>Shyama Prasad Mukherjee Fellow<br>
Inter-University Center for Astronomy and Astrophysics (IUCAA)<br>
Pune, India<div><br></div></div></div>