Dear Erik and Shamim,

I ran into this problem in the past. There are tickets about it:

https://bitbucket.org/einsteintoolkit/tickets/issues/2547/compilation-failur...

https://bitbucket.org/einsteintoolkit/tickets/issues/2115/supermike-ii-fails...

Gabriele

On Wed, Nov 2, 2022 at 8:35 AM Erik Schnetter schnetter@gmail.com wrote:

Shamim

The actual error message is

/home2/mallick/ET4/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(1224): error: invalid type for defaulted constructor vmask(vmask &&) = default;

The code in question is standard C++ code. I don't understand why the compiler would not accept it. I see that you are using an old (2013) version of the Intel compiler; can you try using a newer compiler? This might be a bug in the compiler.

-erik

On Wed, Nov 2, 2022 at 4:38 AM Shamim Haque 1910511 shamims@iiserb.ac.in wrote:

...

Hi,

I am trying to build ET on the institute cluster KANAD at IISER Bhopal.

The compiling stops at the following point, which I am unable to process what could be the issue:

...

COMPILING EinsteinInitialData/Exact/src/metrics/bowl.F COMPILING EinsteinInitialData/Exact/src/metrics/constant_density_star.F Creating /home2/mallick/ET4/Cactus/configs/sim/lib/libthorn_Extract.a Creating /home2/mallick/ET4/Cactus/configs/sim/lib/libthorn_Exact.a make: *** [sim] Error 2

I am attaching the full terminal output and all the scripts (edited as

per the cluster). Any help in this regard will be helpful. Thanks in advance.

...

Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Erik Schnetter schnetter@gmail.com http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Dear Erik, Gabriele and Roland,

Thanks a lot for pointing out the issue. We tried a few things after looking at the tickets and solutions, and each of them landed with different errors. I am listing them below in 3 parts:

*Part A:* Keeping compiler Intel 2013, I tried to add lines -gcc_name and -gxx_name to CFLAGS and CXXFLAGS pointing to gcc/7.3.0. The error while compiling is the following:

*checking for C++ lambda expressions... yeschecking for C++ range-based for statements... noCactus requires a C++11 compiler -- check your C++ compiler and C++ compiler flagsError reconfiguring sim-configmake: *** [sim-config] Error 2*

I have attached the ini file (kanad_et7.ini), cfg file (kanad_et7.cfg) and full terminal output (out_et7.txt) for reference. I am unable to extract the issue at this point. The module gcc/7.3.0 seems to be loading properly using *module load*, as the *which cpp *changes from default cpp to cpp-7.3.0.

*PART B:* The KANAD hpc also has Intel compiler 2017, but the module file for the same is not available, so the default icc/icpc/ifort still remains intel 2013 during attempts to compile ET. I made changes in cfg file accordingly to accommodate intel 2017 but compilation of this ET lands on another issue:

*checking for mawk... nochecking for gawk... gawkchecking whether make sets ${MAKE}... yeschecking whether the C compiler (/opt/intel-17/compilers_and_libraries_2017/linux/bin/intel64/icc -g -xHOST -align -std=gnu99 -Wl,-rpath,/opt/intel-17/compilers_and_libraries_2017/linux/bin/intel64 -Wl,-rpath, /opt/intel-17/compilers_and_libraries_2017/linux/mkl/lib/intel64) works... noconfigure: error: installation or configuration problem: C compiler cannot create executables (see configs/sim/config-data/config.log for details).Error reconfiguring sim-configmake: *** [sim-config] Error 2*

Upon checking the config log file, I found the following:

*Error: Product support for your (Comp-CL) license has expired.License file(s) used were (in this order): 1. Trusted Storage** 2. /opt/intel/licenses/NCOM_L___2HWS-TFKRNMR4_1.lic** 3. /opt/intel/licenses/server.lic** 4. /opt/intel/composer_xe_2013.1.117/licenses/license.lic** 5. /home2/mallick/intel/licenses** 6. /opt/intel-17/compilers_and_libraries_2017.4.196/linux/bin/intel64/../../Licenses** 7. /home2/mallick/Licenses** 8. /Users/Shared/Library/Application Support/Intel/Licenses** 9. /opt/intel-17/compilers_and_libraries_2017.4.196/linux/bin/intel64/*.licPlease refer https://software.intel.com/en-us/faq/purchasing-renewing-upgrading#support-e... https://software.intel.com/en-us/faq/purchasing-renewing-upgrading#support-expiration for more information..icc: error #10052: could not checkout FLEXlm licenseconfigure: failed program was:#line 1094 "configure"#include "confdefs.h"*

Seems like some issue with the license, but I checked with the hpc admin and he says the license should not be any issue. I am unable to figure out the issue here as well. Could this be the problem that my environment default intel compiler is still intel 2013 while compiling? I have attached the cfg file (kanad_et4.cfg), full terminal output (out_et4.out) and config log file (config_et4.log) for reference.

*Part C:* I compiled another ET successfully using the modules gcc-7.3.0, openmpi-3.1.4, FFTW3/3.3.3, gsl/1.16, openssl/1.1.1a, zlib/1.2.8, cmake/3.15.4, libjpeg/1.2.1, HDF5/1.8.10, openmpi/3.1.4.

But there seems to be a repetitive warning while buliding ET, which I am not sure if I should be worried about: */usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/atlas/liblapack.so, may conflict with libgfortran.so.4*

Please let me know if I should consider changing something to get rid of this warning, I have attached ini file (kanad_et8.ini), cfg file (kanad_et8.cfg) and full terminal output (out_et8.txt) for reference.

I am currently trying to run helloworld, but it stops as it cannot find openmpi in computing nodes. Later, I checked, openmpi/3.1.4 actually does not show up in the module list of those computing nodes. I am checking with the hpc admin about this issue.

Now, having got this compiled successfully, should I continue to pursue compiling ET with intel compilers? Though I am still not sure if this ET (with gcc-7.3) will show up any errors in future as I aim to work on binary neutron star merger simulations. Please let me know your thoughts on this.

Thanks and Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal

On Wed, Nov 2, 2022 at 9:27 PM Roland Haas rhaas@illinois.edu wrote:

Hello all,

First: thank you for including all the output and configuration files. This helps a lot!

...

The actual error message is

/home2/mallick/ET4/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(1224):

...

error: invalid type for defaulted constructor vmask(vmask &&) = default;

The code in question is standard C++ code. I don't understand why the compiler would not accept it. I see that you are using an old (2013) version of the Intel compiler; can you try using a newer compiler? This might be a bug in the compiler.

In most cases of C++ errors involving the intel compiler that I have seen in the recent past, the issue was the that the Intel C++ compiler relies on the C++ template library of the GNU compiler (g++). It needs to be told to use a new enough library usually.

I would try and do the following:

• find a g++ compiler that it at least gcc 6
• add -gxx-name PATH-TO-NEW-G++ to your CXXFLAGS
• compile from scratch

See eg:

https://www.einsteintoolkit.org/about/releases/ET_2022_05_announcement.html

or

https://bitbucket.org/einsteintoolkit/tickets/issues/2547/compilation-failur...

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

Hello Roland,

Thanks for the comments on the issues. I have attached the config.log file for Part A, and I'll see what can be done about Part B.

For Part C, I found a LAPACK library in our HPC, and the compilation process was completed without that warning. The Helloworld is also executed correctly.

Currently, with this ET, I am facing issue on executing the gallery BNSM simulation on 2 nodes. The info in the command line goes as follows:

*./simfactory/bin/sim create-submit nsns_p32_t24_8 --procs=32 --ppn=16 --num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par --walltime=25:00:00*

In the out file, the error is:

*+ mpiexec -n 32 -npernode 16 /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim -L 3 /home2/mallick/simulations2/nsns_p32_t24_11/output-0000/nsnstohmns1.par--------------------------------------------------------------------------Your job has requested more processes than the ppr forthis topology can support: App: /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim Number of procs: 32 PPR: 16:nodePlease revise the conflict and try again.--------------------------------------------------------------------------Simfactory Done at date: Mon 14 Nov 2022 07:11:57 PM IST*

The simulations seem to work fine with 1 node, that is, --procs set to 16,8 or 4 keeping --ppn to 16. I assume the mpiexec command in runscript needs to be updated to provide clarity to the mpi library while using more than 1 node.

I need some help here, I don't know how to proceed at this point. I am attaching the run, ini, cfg, sub script used for this ET and the output file for reference. Thanks in advance for all the help.

Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal

ᐧ

On Mon, Nov 14, 2022 at 8:22 PM Roland Haas rhaas@illinois.edu wrote:

Hello Shamim Haque,

...

*Part A:* Keeping compiler Intel 2013, I tried to add lines -gcc_name and -gxx_name to CFLAGS and CXXFLAGS pointing to gcc/7.3.0. The error while compiling

is

...

the following:

*checking for C++ lambda expressions... yeschecking for C++ range-based

for

...

statements... noCactus requires a C++11 compiler -- check your C++

compiler

...

and C++ compiler flagsError reconfiguring sim-configmake: ***

[sim-config]

...

Error 2*

This still sounds like a compiler incompatibility. Somehow Cactus does not detect support for range based for statements, which however Intel 2013 claims to support in (search for "range-based", it is item N2930)

https://www.intel.com/content/www/us/en/developer/articles/technical/c0x-fea...

What does the file config.log (that autoconf points you to for detailed error messages) contain? It usually is something like configs/sim/config-data/config.log . The options used for CXXFLAGS (if those are the ones used) look fine to me.

...

*PART B:*

...

*Error: Product support for your (Comp-CL) license has expired.License file(s) used were (in this order): 1. Trusted Storage** 2.

Yes, this is a license issue. Note that you may still require -gxx-name options even for newer Intel compilers (they may default to the system g++ and system STL otherwise).

...

*Part C:* I compiled another ET successfully using the modules gcc-7.3.0, openmpi-3.1.4, FFTW3/3.3.3, gsl/1.16, openssl/1.1.1a, zlib/1.2.8, cmake/3.15.4, libjpeg/1.2.1, HDF5/1.8.10, openmpi/3.1.4.

But there seems to be a repetitive warning while buliding ET, which I am not sure if I should be worried about: */usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/atlas/liblapack.so, may conflict with libgfortran.so.4*

Basically: /usr/lib64/atlas/liblapack.so (the system ATLAS library) has been compiled with at version of gfortran much older than the one you are using. This can be fail in particular when involving strings being passed to Fortran code.

...

Please let me know if I should consider changing something to get rid of this warning, I have attached ini file (kanad_et8.ini), cfg file (kanad_et8.cfg) and full terminal output (out_et8.txt) for reference.

You may need to set:

LAPACK_DIR = BUILD BLAS_DIR = BUILD

to force the EinsteinToolkit to build its own (slow, but we do not rely on BLAS / LAPACK for speed) versions of LAPACK and BLAS (or you can try using OpenBLAS which is faster, but as said, speed of those two is not really relevant for typical ET simulations).

...

Now, having got this compiled successfully, should I continue to pursue compiling ET with intel compilers? Though I am still not sure if this ET (with gcc-7.3) will show up any errors in future as I aim to work on

binary

...

neutron star merger simulations. Please let me know your thoughts on

this.

Historically we did see slightly faster code with the Intel compiler. I suspect that similar speeds can be reached using GNU compilers by now though if one sets -ffast-math and similar options (that Intel defaults to) in CFLAGS and CXXFLAGS (Fortran has some of those optimizations allowed by the language already so it does not do quite so much for Fortran code).

See eg: https://gcc.gnu.org/wiki/FloatingPointMath for an explanation of the compromises this involves.

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

Hello Shamim Haque,

Not all errors in config.log indicate an actual failure, some just show up b/c a configure test fails and a workaround is implemented in Cactus.

Unfortunately the intel compiler have both a minimum and a maximum g++ (and STL) version that can be used. The old intel 2013 compiler may not support g++ 7.0.

While I could not find the actual information for intel 2013 (Intel hides this quite well), there is this stackoverflow post:

https://stackoverflow.com/questions/29371301/intel-c-compiler-what-is-highes...

that reports tha the *newest* version of gcc supported by icpc 15.0.1 (2014-10-23) is gcc 4.9 so the even older intel 13 compiler is not going to support any newer gcc.

Eg this

https://www.cism.ucl.ac.be/Services/Formations/ICS/ics_2013.0.028/composer_x...

which is for an intel 2013 states that

Support for C++11 features in gcc 4.6 and 4.7 headers

so quite likely you need a newer Intel compiler than Intel 2013 (9 years old by now after all).

From the 2017 release notes

https://www.intel.com/content/www/us/en/developer/articles/release-notes/int...

it would seem that it does support gcc 6:

--8<-- Linux Developer tools component installed, including gcc, g++ and related tools

gcc versions 4.3 - 6 supported --8<--

Yours, Roland

A small correction to my previous email, the config log file for part A is the one attached here, which is compiling attempt with Intel 2013 + gcc-7.3.0. The one in my previous mail is from compiling attempt with Intel 2013 + gcc-4.9.3.

However, the errors in the config files look identical.

Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal

ᐧ

On Tue, Nov 15, 2022 at 2:30 AM Shamim Haque 1910511 shamims@iiserb.ac.in wrote:

...

Hello Roland,

Thanks for the comments on the issues. I have attached the config.log file for Part A, and I'll see what can be done about Part B.

For Part C, I found a LAPACK library in our HPC, and the compilation process was completed without that warning. The Helloworld is also executed correctly.

Currently, with this ET, I am facing issue on executing the gallery BNSM simulation on 2 nodes. The info in the command line goes as follows:

*./simfactory/bin/sim create-submit nsns_p32_t24_8 --procs=32 --ppn=16 --num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par --walltime=25:00:00*

In the out file, the error is:

*+ mpiexec -n 32 -npernode 16 /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim -L 3 /home2/mallick/simulations2/nsns_p32_t24_11/output-0000/nsnstohmns1.par--------------------------------------------------------------------------Your job has requested more processes than the ppr forthis topology can support: App: /home2/mallick/simulations2/nsns_p32_t24_11/SIMFACTORY/exe/cactus_sim Number of procs: 32 PPR: 16:nodePlease revise the conflict and try again.--------------------------------------------------------------------------Simfactory Done at date: Mon 14 Nov 2022 07:11:57 PM IST*

The simulations seem to work fine with 1 node, that is, --procs set to 16,8 or 4 keeping --ppn to 16. I assume the mpiexec command in runscript needs to be updated to provide clarity to the mpi library while using more than 1 node.

I need some help here, I don't know how to proceed at this point. I am attaching the run, ini, cfg, sub script used for this ET and the output file for reference. Thanks in advance for all the help.

Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal

ᐧ

On Mon, Nov 14, 2022 at 8:22 PM Roland Haas rhaas@illinois.edu wrote:

...

Hello Shamim Haque,

...

*Part A:* Keeping compiler Intel 2013, I tried to add lines -gcc_name and

-gxx_name

...

to CFLAGS and CXXFLAGS pointing to gcc/7.3.0. The error while

compiling is

...

the following:

*checking for C++ lambda expressions... yeschecking for C++ range-based

for

...

statements... noCactus requires a C++11 compiler -- check your C++

compiler

...

and C++ compiler flagsError reconfiguring sim-configmake: ***

[sim-config]

...

Error 2*

This still sounds like a compiler incompatibility. Somehow Cactus does not detect support for range based for statements, which however Intel 2013 claims to support in (search for "range-based", it is item N2930)

https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/develope...

What does the file config.log (that autoconf points you to for detailed error messages) contain? It usually is something like configs/sim/config-data/config.log . The options used for CXXFLAGS (if those are the ones used) look fine to me.

...

*PART B:*

...

*Error: Product support for your (Comp-CL) license has expired.License file(s) used were (in this order): 1. Trusted Storage** 2.

Yes, this is a license issue. Note that you may still require -gxx-name options even for newer Intel compilers (they may default to the system g++ and system STL otherwise).

...

*Part C:* I compiled another ET successfully using the modules gcc-7.3.0, openmpi-3.1.4, FFTW3/3.3.3, gsl/1.16, openssl/1.1.1a, zlib/1.2.8, cmake/3.15.4, libjpeg/1.2.1, HDF5/1.8.10, openmpi/3.1.4.

But there seems to be a repetitive warning while buliding ET, which I am not sure if I should be worried about: */usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/atlas/liblapack.so, may conflict with libgfortran.so.4*

Basically: /usr/lib64/atlas/liblapack.so (the system ATLAS library) has been compiled with at version of gfortran much older than the one you are using. This can be fail in particular when involving strings being passed to Fortran code.

...

Please let me know if I should consider changing something to get rid of this warning, I have attached ini file (kanad_et8.ini), cfg file (kanad_et8.cfg) and full terminal output (out_et8.txt) for reference.

You may need to set:

LAPACK_DIR = BUILD BLAS_DIR = BUILD

to force the EinsteinToolkit to build its own (slow, but we do not rely on BLAS / LAPACK for speed) versions of LAPACK and BLAS (or you can try using OpenBLAS which is faster, but as said, speed of those two is not really relevant for typical ET simulations).

...

Now, having got this compiled successfully, should I continue to pursue compiling ET with intel compilers? Though I am still not sure if this ET (with gcc-7.3) will show up any errors in future as I aim to work on

binary

...

neutron star merger simulations. Please let me know your thoughts on

this.

Historically we did see slightly faster code with the Intel compiler. I suspect that similar speeds can be reached using GNU compilers by now though if one sets -ffast-math and similar options (that Intel defaults to) in CFLAGS and CXXFLAGS (Fortran has some of those optimizations allowed by the language already so it does not do quite so much for Fortran code).

See eg: https://urldefense.com/v3/__https://gcc.gnu.org/wiki/FloatingPointMath__;!!D... for an explanation of the compromises this involves.

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4p6GfR3s9DU1DFu1e4... .

Hello Shamim,

Thanks for pointing out the issue. I'll check with HPC admins to fix the Intel-17 or install a better version. Could you please help me with the issue regarding running BNSM on more than 1 node? The issue is explained in my previous email in reply to your suggestions on problems divided into Part A, B and C.

That is unfortunately quite tricky to do. Each cluster handles this a bit differently. My best suggestion is to first look for an example of a hybrid job (MPI+OpenMP) that the cluster admins hopefully provide.

Next you must construct a submitscript (template) that matches their example (more or less).

You can try these out by submitting test jobs and looking at the file:

<basedir>/<jobname>/output-0000/SIMFACTORY/SubmitScript

which has all the replacements done.

For testing you may want to set the "submit" option in the machine ini file to just "echo 42" or so so that no actual job is submitted.

I will be, unfortunately, quite busy until after the ET release (this week) and most likely also next week, so cannot really promise to be able look into this very deeply.

My best suggestion is to track down an OpenMP+MPI example in the cluster documentation and then tweak your SubmitScript (foo.sub) file until you have something that, when provided the correct options, matches their example.

The error that you received basically says that your submitscript asked for more resources than available, but this can be triggered by a number of things including configuration options set by the admins.

Yours, Roland