Hello Roland,

Thanks for the comments on the issues. I have attached the config.log file for Part A, and I'll see what can be done about Part B. 

For Part C, I found a LAPACK library in our HPC, and the compilation process was completed without that warning. The Helloworld is also executed correctly. 

Currently, with this ET, I am facing issue on executing the gallery BNSM simulation on 2 nodes. The info in the command line goes as follows:

./simfactory/bin/sim create-submit nsns_p32_t24_8 --procs=32 --ppn=16 --num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par --walltime=25:00:00

In the out file, the error is:

+ mpiexec -n 32 -npernode 16 /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim -L 3 /home2/mallick/simulations2/nsns_p32_t24_11/output-0000/nsnstohmns1.par
--------------------------------------------------------------------------
Your job has requested more processes than the ppr for
this topology can support:
  App: /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim
  Number of procs:  32
  PPR: 16:node
Please revise the conflict and try again.
--------------------------------------------------------------------------
Simfactory Done at date: Mon 14 Nov 2022 07:11:57 PM IST

The simulations seem to work fine with 1 node, that is, --procs set to 16,8 or 4 keeping --ppn to 16. I assume the mpiexec command in runscript needs to be updated to provide clarity to the mpi library while using more than 1 node.

I need some help here, I don't know how to proceed at this point. I am attaching the run, ini, cfg, sub script used for this ET and the output file for reference. Thanks in advance for all the help.

Regards

Shamim Haque

Senior Research Fellow (SRF)

Department of Physics

IISER Bhopal

ᐧ

On Mon, Nov 14, 2022 at 8:22 PM Roland Haas <rhaas@illinois.edu> wrote:

Hello Shamim Haque,

> *Part A:*
> Keeping compiler Intel 2013, I tried to add lines -gcc_name and -gxx_name
> to CFLAGS and CXXFLAGS pointing to gcc/7.3.0. The error while compiling is
> the following:
>
>
>
>
>
> *checking for C++ lambda expressions... yeschecking for C++ range-based for
> statements... noCactus requires a C++11 compiler -- check your C++ compiler
> and C++ compiler flagsError reconfiguring sim-configmake: *** [sim-config]
> Error 2*

This still sounds like a compiler incompatibility. Somehow Cactus does
not detect support for range based for statements, which however Intel
2013 claims to support in (search for "range-based", it is  item N2930)

https://www.intel.com/content/www/us/en/developer/articles/technical/c0x-features-supported-by-intel-c-compiler.html

What does the file config.log (that autoconf points you to for detailed
error messages) contain? It usually is something like
configs/sim/config-data/config.log . The options used for CXXFLAGS (if
those are the ones used) look fine to me.

> *PART B:*

> *Error: Product support for your (Comp-CL) license has expired.License
> file(s) used were (in this order):    1.  Trusted Storage**  2.

Yes, this is a license issue. Note that you may still require -gxx-name
options even for newer Intel compilers (they may default to the system
g++ and system STL otherwise).

> *Part C:*
> I compiled another ET successfully using the modules gcc-7.3.0,
> openmpi-3.1.4, FFTW3/3.3.3, gsl/1.16, openssl/1.1.1a, zlib/1.2.8,
> cmake/3.15.4, libjpeg/1.2.1, HDF5/1.8.10,  openmpi/3.1.4.
>
> But there seems to be a repetitive warning while buliding ET, which I am
> not sure if I should be worried about:
> */usr/bin/ld: warning: libgfortran.so.3, needed by
> /usr/lib64/atlas/liblapack.so, may conflict with libgfortran.so.4*

Basically: /usr/lib64/atlas/liblapack.so (the system ATLAS library) has
been compiled with at version of gfortran much older than the one you
are using. This can be fail in particular when involving strings being
passed to Fortran code.

> Please let me know if I should consider changing something to get rid of
> this warning, I have attached ini file (kanad_et8.ini), cfg file
> (kanad_et8.cfg) and full terminal output (out_et8.txt) for reference.

You may need to set:

LAPACK_DIR = BUILD
BLAS_DIR = BUILD

to force the EinsteinToolkit to build its own (slow, but we do not rely
on BLAS / LAPACK for speed) versions of LAPACK and BLAS (or you can try
using OpenBLAS which is faster, but as said, speed of those two is not
really relevant for typical ET simulations).

> Now, having got this compiled successfully, should I continue to pursue
> compiling ET with intel compilers? Though I am still not sure if this ET
> (with gcc-7.3) will show up any errors in future as I aim to work on binary
> neutron star merger simulations. Please let me know your thoughts on this.

Historically we did see slightly faster code with the Intel compiler. I
suspect that similar speeds can be reached using GNU compilers by now
though if one sets -ffast-math and similar options (that Intel
defaults to) in CFLAGS and CXXFLAGS (Fortran has some of those
optimizations allowed by the language already so it does not do quite
so much for Fortran code).

See eg: https://gcc.gnu.org/wiki/FloatingPointMath for an explanation
of the compromises this involves.

Yours,
Roland

--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .