<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="color:rgb(0,0,0)">Hello Roland, </div><div class="gmail_default" style="color:rgb(0,0,0)"><br></div><div class="gmail_default" style="color:rgb(0,0,0)">Thanks for pointing out the issue. I'll check with HPC admins to fix the Intel-17 or install a better version. Could you please help me with the issue regarding running BNSM on more than 1 node? The issue is explained in my previous email in reply to your suggestions on problems divided into Part A, B and C. </div><div class="gmail_default" style="color:rgb(0,0,0)"><br></div><div class="gmail_default" style="color:rgb(0,0,0)">Regards</div><div class="gmail_default" style="color:rgb(0,0,0)">Shamim</div><div class="gmail_default" style="color:rgb(0,0,0)"><br></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><a href="https://streaklinks.com/BR5Cd08uLuXq6QxdRQF3TX0k/https%3A%2F%2Fwww.cism.ucl.ac.be%2FServices%2FFormations%2FICS%2Fics_2013.0.028%2Fcomposer_xe_2013.1.117%2FDocumentation%2Fen_US%2FRelease_Notes_C_2013_L_EN.pdf" rel="noreferrer" target="_blank"><br></a>> > For Part C, I found a LAPACK library in our HPC, and the compilation<br>
> > process was completed without that warning. The Helloworld is also executed<br>
> > correctly.<br>
> ><br>
> > Currently, with this ET, I am facing issue on executing the gallery BNSM<br>
> > simulation on 2 nodes. The info in the command line goes as follows:<br>
> ><br>
> > *./simfactory/bin/sim create-submit nsns_p32_t24_8 --procs=32 --ppn=16<br>
> > --num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par<br>
> > --walltime=25:00:00*<br>
> ><br>
> > In the out file, the error is:<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > *+ mpiexec -n 32 -npernode 16<br>
> > /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim -L 3<br>
> > /home2/mallick/simulations2/nsns_p32_t24_11/output-0000/nsnstohmns1.par--------------------------------------------------------------------------Your<br>
> > job has requested more processes than the ppr forthis topology can<br>
> > support: App:<br>
> > /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim<br>
> > Number of procs: 32 PPR: 16:nodePlease revise the conflict and try<br>
> > again.--------------------------------------------------------------------------Simfactory<br>
> > Done at date: Mon 14 Nov 2022 07:11:57 PM IST*<br>
> ><br>
> > The simulations seem to work fine with 1 node, that is, --procs set to<br>
> > 16,8 or 4 keeping --ppn to 16. I assume the mpiexec command in runscript<br>
> > needs to be updated to provide clarity to the mpi library while using more<br>
> > than 1 node.<br>
> ><br>
> > I need some help here, I don't know how to proceed at this point. I am<br>
> > attaching the run, ini, cfg, sub script used for this ET and the output<br>
> > file for reference. Thanks in advance for all the help.<br>
> ><br>
> > Regards<br>
> > Shamim Haque<br>
> > Senior Research Fellow (SRF)<br>
> > Department of Physics<br>
> > IISER Bhopal<br>
> ><br>
> > ᐧ<br>
> ><br>
> > On Mon, Nov 14, 2022 at 8:22 PM Roland Haas <<a href="mailto:rhaas@illinois.edu" target="_blank">rhaas@illinois.edu</a>> wrote:<br>
> ><br>
> >> Hello Shamim Haque,<br>
> >><br>
> >> > *Part A:*<br>
> >> > Keeping compiler Intel 2013, I tried to add lines -gcc_name and<br>
> >> -gxx_name<br>
> >> > to CFLAGS and CXXFLAGS pointing to gcc/<a href="https://streaklinks.com/BR5Cd08_vwoS01JRmAmyuuGa/http%3A%2F%2F7.3.0." rel="noreferrer" target="_blank">7.3.0.</a> The error while<br>
> >> compiling is<br>
> >> > the following:<br>
> >> ><br>
> >> ><br>
> >> ><br>
> >> ><br>
> >> ><br>
> >> > *checking for C++ lambda expressions... yeschecking for C++ range-based<br>
> >> for<br>
> >> > statements... noCactus requires a C++11 compiler -- check your C++<br>
> >> compiler<br>
> >> > and C++ compiler flagsError reconfiguring sim-configmake: ***<br>
> >> [sim-config]<br>
> >> > Error 2*<br>
> >><br>
> >> This still sounds like a compiler incompatibility. Somehow Cactus does<br>
> >> not detect support for range based for statements, which however Intel<br>
> >> 2013 claims to support in (search for "range-based", it is item N2930)<br>
> >><br>
> >><br>
> >> <a href="https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/developer/articles/technical/c0x-features-supported-by-intel-c-compiler.html__;!!DZ3fjg!4p6GfR3s9DU1DFu1e47xeUg3PBbxUi312fkmUSjyr8brzIS1vwSemU9s5aD35GvJj6eOhHeniZM3HGkosXBN$" rel="noreferrer" target="_blank">https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/developer/articles/technical/c0x-features-supported-by-intel-c-compiler.html__;!!DZ3fjg!4p6GfR3s9DU1DFu1e47xeUg3PBbxUi312fkmUSjyr8brzIS1vwSemU9s5aD35GvJj6eOhHeniZM3HGkosXBN$</a> <br>
> >><br>
> >> What does the file config.log (that autoconf points you to for detailed<br>
> >> error messages) contain? It usually is something like<br>
> >> configs/sim/config-data/config.log . The options used for CXXFLAGS (if<br>
> >> those are the ones used) look fine to me.<br>
> >><br>
> >> > *PART B:*<br>
> >><br>
> >> > *Error: Product support for your (Comp-CL) license has expired.License<br>
> >> > file(s) used were (in this order): 1. Trusted Storage** 2.<br>
> >><br>
> >> Yes, this is a license issue. Note that you may still require -gxx-name<br>
> >> options even for newer Intel compilers (they may default to the system<br>
> >> g++ and system STL otherwise).<br>
> >><br>
> >> > *Part C:*<br>
> >> > I compiled another ET successfully using the modules gcc-7.3.0,<br>
> >> > openmpi-3.1.4, FFTW3/3.3.3, gsl/1.16, openssl/1.1.1a, zlib/1.2.8,<br>
> >> > cmake/3.15.4, libjpeg/1.2.1, HDF5/1.8.10, openmpi/<a href="https://streaklinks.com/BR5Cd08MjEDU_beQoAFbaxq9/http%3A%2F%2F3.1.4." rel="noreferrer" target="_blank" hspace="streak-track">3.1.4.</a><br>
> >> ><br>
> >> > But there seems to be a repetitive warning while buliding ET, which I am<br>
> >> > not sure if I should be worried about:<br>
> >> > */usr/bin/ld: warning: libgfortran.so.3, needed by<br>
> >> > /usr/lib64/atlas/liblapack.so, may conflict with libgfortran.so.4*<br>
> >><br>
> >> Basically: /usr/lib64/atlas/liblapack.so (the system ATLAS library) has<br>
> >> been compiled with at version of gfortran much older than the one you<br>
> >> are using. This can be fail in particular when involving strings being<br>
> >> passed to Fortran code.<br>
> >><br>
> >> > Please let me know if I should consider changing something to get rid of<br>
> >> > this warning, I have attached ini file (kanad_et8.ini), cfg file<br>
> >> > (kanad_et8.cfg) and full terminal output (out_et8.txt) for reference.<br>
> >><br>
> >> You may need to set:<br>
> >><br>
> >> LAPACK_DIR = BUILD<br>
> >> BLAS_DIR = BUILD<br>
> >><br>
> >> to force the EinsteinToolkit to build its own (slow, but we do not rely<br>
> >> on BLAS / LAPACK for speed) versions of LAPACK and BLAS (or you can try<br>
> >> using OpenBLAS which is faster, but as said, speed of those two is not<br>
> >> really relevant for typical ET simulations).<br>
> >><br>
> >> > Now, having got this compiled successfully, should I continue to pursue<br>
> >> > compiling ET with intel compilers? Though I am still not sure if this ET<br>
> >> > (with gcc-7.3) will show up any errors in future as I aim to work on<br>
> >> binary<br>
> >> > neutron star merger simulations. Please let me know your thoughts on<br>
> >> this.<br>
> >><br>
> >> Historically we did see slightly faster code with the Intel compiler. I<br>
> >> suspect that similar speeds can be reached using GNU compilers by now<br>
> >> though if one sets -ffast-math and similar options (that Intel<br>
> >> defaults to) in CFLAGS and CXXFLAGS (Fortran has some of those<br>
> >> optimizations allowed by the language already so it does not do quite<br>
> >> so much for Fortran code).<br>
> >><br>
> >> See eg: <a href="https://urldefense.com/v3/__https://gcc.gnu.org/wiki/FloatingPointMath__;!!DZ3fjg!4p6GfR3s9DU1DFu1e47xeUg3PBbxUi312fkmUSjyr8brzIS1vwSemU9s5aD35GvJj6eOhHeniZM3HJ06ogmy$" rel="noreferrer" target="_blank" hspace="streak-track">https://urldefense.com/v3/__https://gcc.gnu.org/wiki/FloatingPointMath__;!!DZ3fjg!4p6GfR3s9DU1DFu1e47xeUg3PBbxUi312fkmUSjyr8brzIS1vwSemU9s5aD35GvJj6eOhHeniZM3HJ06ogmy$</a> for an explanation<br>
> >> of the compromises this involves.<br>
> >><br>
> >> Yours,<br>
> >> Roland<br>
> >><br>
> >> --<br>
> >> My email is as private as my paper mail. I therefore support encrypting<br>
> >> and signing email messages. Get my PGP key from <a href="https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4p6GfR3s9DU1DFu1e47xeUg3PBbxUi312fkmUSjyr8brzIS1vwSemU9s5aD35GvJj6eOhHeniZM3HE00Tvms$" rel="noreferrer" target="_blank" hspace="streak-track">https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4p6GfR3s9DU1DFu1e47xeUg3PBbxUi312fkmUSjyr8brzIS1vwSemU9s5aD35GvJj6eOhHeniZM3HE00Tvms$</a> .<br>
> >><br>
> ><br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank" hspace="streak-track">http://pgp.mit.edu</a> .<br>
</blockquote></div></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=3bab59bc-6915-4c63-89d6-9b5af59a1a34"><font color="#ffffff" size="1">ᐧ</font></div>