[Users] Issue with compiling ET on cluster

Mon Nov 14 15:00:28 CST 2022

Hello Roland,

Thanks for the comments on the issues. I have attached the config.log file
for Part A, and I'll see what can be done about Part B.

For Part C, I found a LAPACK library in our HPC, and the compilation
process was completed without that warning. The Helloworld is also executed
correctly.

Currently, with this ET, I am facing issue on executing the gallery BNSM
simulation on 2 nodes. The info in the command line goes as follows:

*./simfactory/bin/sim create-submit nsns_p32_t24_8 --procs=32 --ppn=16
--num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par
--walltime=25:00:00*

In the out file, the error is:

*+ mpiexec -n 32 -npernode 16
/home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim -L 3
/home2/mallick/simulations2/nsns_p32_t24_11/output-0000/nsnstohmns1.par--------------------------------------------------------------------------Your
job has requested more processes than the ppr forthis topology can
support:  App:
/home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim
Number of procs:  32  PPR: 16:nodePlease revise the conflict and try
again.--------------------------------------------------------------------------Simfactory
Done at date: Mon 14 Nov 2022 07:11:57 PM IST*

The simulations seem to work fine with 1 node, that is, --procs set to 16,8
or 4 keeping --ppn to 16. I assume the mpiexec command in runscript needs
to be updated to provide clarity to the mpi library while using more than 1
node.

I need some help here, I don't know how to proceed at this point. I am
attaching the run, ini, cfg, sub script used for this ET and the output
file for reference. Thanks in advance for all the help.

Regards
Shamim Haque
Senior Research Fellow (SRF)
Department of Physics
IISER Bhopal

ᐧ

On Mon, Nov 14, 2022 at 8:22 PM Roland Haas <rhaas at illinois.edu> wrote:

> Hello Shamim Haque,
>
> > *Part A:*
> > Keeping compiler Intel 2013, I tried to add lines -gcc_name and -gxx_name
> > to CFLAGS and CXXFLAGS pointing to gcc/7.3.0. The error while compiling
> is
> > the following:
> >
> >
> >
> >
> >
> > *checking for C++ lambda expressions... yeschecking for C++ range-based
> for
> > statements... noCactus requires a C++11 compiler -- check your C++
> compiler
> > and C++ compiler flagsError reconfiguring sim-configmake: ***
> [sim-config]
> > Error 2*
>
> This still sounds like a compiler incompatibility. Somehow Cactus does
> not detect support for range based for statements, which however Intel
> 2013 claims to support in (search for "range-based", it is  item N2930)
>
>
> https://www.intel.com/content/www/us/en/developer/articles/technical/c0x-features-supported-by-intel-c-compiler.html
>
> What does the file config.log (that autoconf points you to for detailed
> error messages) contain? It usually is something like
> configs/sim/config-data/config.log . The options used for CXXFLAGS (if
> those are the ones used) look fine to me.
>
> > *PART B:*
>
> > *Error: Product support for your (Comp-CL) license has expired.License
> > file(s) used were (in this order):    1.  Trusted Storage**  2.
>
> Yes, this is a license issue. Note that you may still require -gxx-name
> options even for newer Intel compilers (they may default to the system
> g++ and system STL otherwise).
>
> > *Part C:*
> > I compiled another ET successfully using the modules gcc-7.3.0,
> > openmpi-3.1.4, FFTW3/3.3.3, gsl/1.16, openssl/1.1.1a, zlib/1.2.8,
> > cmake/3.15.4, libjpeg/1.2.1, HDF5/1.8.10,  openmpi/3.1.4.
> >
> > But there seems to be a repetitive warning while buliding ET, which I am
> > not sure if I should be worried about:
> > */usr/bin/ld: warning: libgfortran.so.3, needed by
> > /usr/lib64/atlas/liblapack.so, may conflict with libgfortran.so.4*
>
> Basically: /usr/lib64/atlas/liblapack.so (the system ATLAS library) has
> been compiled with at version of gfortran much older than the one you
> are using. This can be fail in particular when involving strings being
> passed to Fortran code.
>
> > Please let me know if I should consider changing something to get rid of
> > this warning, I have attached ini file (kanad_et8.ini), cfg file
> > (kanad_et8.cfg) and full terminal output (out_et8.txt) for reference.
>
> You may need to set:
>
> LAPACK_DIR = BUILD
> BLAS_DIR = BUILD
>
> to force the EinsteinToolkit to build its own (slow, but we do not rely
> on BLAS / LAPACK for speed) versions of LAPACK and BLAS (or you can try
> using OpenBLAS which is faster, but as said, speed of those two is not
> really relevant for typical ET simulations).
>
> > Now, having got this compiled successfully, should I continue to pursue
> > compiling ET with intel compilers? Though I am still not sure if this ET
> > (with gcc-7.3) will show up any errors in future as I aim to work on
> binary
> > neutron star merger simulations. Please let me know your thoughts on
> this.
>
> Historically we did see slightly faster code with the Intel compiler. I
> suspect that similar speeds can be reached using GNU compilers by now
> though if one sets -ffast-math and similar options (that Intel
> defaults to) in CFLAGS and CXXFLAGS (Fortran has some of those
> optimizations allowed by the language already so it does not do quite
> so much for Fortran code).
>
> See eg: https://gcc.gnu.org/wiki/FloatingPointMath for an explanation
> of the compromises this involves.
>
> Yours,
> Roland
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kanad.sub
Type: text/x-microdvd
Size: 766 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kanad.cfg
Type: application/octet-stream
Size: 4142 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kanad.ini
Type: application/octet-stream
Size: 2907 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0005.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nsns_p32_t24_11.out
Type: application/octet-stream
Size: 2317 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0006.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: text/x-log
Size: 62333 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0003.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kanad.run
Type: application/octet-stream
Size: 1959 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20221115/563f1195/attachment-0007.obj 