Hello Shamim Haque,
Great. Glad that we could help you.
Yours, Roland
Dear Roland, Steve and Frank,
I am able to compile ETK properly with GNU compiler 8.3.0. The simulations are also running successfully.
For Intel, the licence issue was cleared after using -j1, but there are a few issues with other libraries, so I have stopped exploring this route right now, maybe I'll try it later.
Thank you for all the help.
Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal
ᐧ
On Tue, Jun 6, 2023 at 6:56 PM Roland Haas rhaas@illinois.edu wrote:
Hello Shamim Haque,
"icpc unable to checkout FLEXlm license" is caused by the compiler being unable to contact the license server. This is unrelated to the C++ compilation issue really.
To be sure:
the only thing you did was add "-gxx-name=/opt/ohpc/pub/compiler/gcc/8.3.0/bin/g++" was it? In particular you did not add / remove any "module load" commands (e.g., added a module load gcc after an existing module load intel)? I am asking b/c module load gcc may unload the intel compiler module and unset the environment variable that the Intel compiler uses to determine how to contact the license server.
you may also try to compile with just "make -j1" or "simfactory/bin/sim build -j1" to use only a single compiler instance in case the number of compiler licenses available is limited or the license server overwhelmed by too many concurrent compilations (by you or other users of the cluster)
The issue with fftw3 and using gcc is that fftw3 (you are using a system provided fftw3? Either from a module or otherwise provided by the admin) has been compiled with the Intel compiler and thus cannot be used with the GNU compiler. You an set "FFTW_DIR = BUILD" to make the Einstein Toolkit build a copy from scratch using an included tar file. Not that this may be the case for other libraries as well. In particular you usually need to ensure that the MPI library used was compiled with the same compiler you use to compile the simulation code.
Yours, Roland
Hello Roland,
Thank you for the reply.
I pointed -gxx-name to gcc/8.3.0, which resulted in an error related to
the
"icpc unable to checkout FLEXlm license". I also tried to use a different intel compiler (and point -gxx-name to gcc/11.1.0) to check if the error can go away, but it remains the same. I communicated this to
the
HPC admin and awaiting their response; please let me know if I should fix something in my configuration. I have attached the out file (intel_compile.out), config log (intel_config.log), and optionlist (intel.cfg) for reference.
Meanwhile, I also attempted to configure ET with gcc/11.1.0, which
resulted
in some issues regarding the FFTW3. I have contacted the admins regarding this, but please let me know if it is something to be fixed within the optionlist. I have attached the out file (gcc_compile.out), config log (gcc_config.log), and optionlist (gcc.cfg) for reference.
The error in the output file says the following:
*Formaline: is still running, e.g. the final link stage.)/usr/bin/ld: warning: libimf.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: libsvml.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not
found
(try using -rpath or -rpath-link)/usr/bin/ld: warning: libirng.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using
-rpath
or -rpath-link)/usr/bin/ld: warning: libintlc.so.5, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath
or
-rpath-link)/usr/bin/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `__intel_cpu_features_init_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `__svml_sincos2'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `_intel_fast_memset'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to
`__intel_cpu_feature_indicator_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so:
undefined reference to `_intel_fast_memmove'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so:
undefined
reference to `_intel_fast_memcpy'*
Thanks in advance for all the help.
Regards Shamim
ᐧ
On Fri, Jun 2, 2023 at 1:31 AM Roland Haas rhaas@illinois.edu wrote:
Hello Shamim Haque,
This most likely is caused by not having a full C++11 software stack enabled.
The Intel compiler always uses the C++ STL library of the GNU g++ compiler and must be matched up with a compatible g++ compiler's STL to correctly support C++11.
This ticket (the last of many, hopefully) contains details:
https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/is...
Basically, the fix is to find a gcc compiler at least version 6 on your cluster, then pass the option
-gxx-name PATH-TO-GXX
to icpc in the CXXFLAGS option list variable (see eg
https://urldefense.com/v3/__https://bitbucket.org/simfactory/simfactory2/src...
).
If there is no gcc compiler version 6 or newer installed in the system, you will have to contact the administrators and ask them about it.
The demo code in the description of
https://urldefense.com/v3/__https://bitbucket.org/cactuscode/cactus/pull-req... may help
demonstrate to the admins what the issue is.
Yours, Roland
Hello All,
I have made little progress on compiling ETK on this cluster, as
the
admins
allowed me to compile ETK on the login node. I am finding new errors
and
am
unable to debug the issue. I am trying to configure ETK using
Intel19,
and
the error says the following:
*/home/shamims.iiserb/ET_debug/Cactus/arrangements/Carpet/CarpetMask/src/mask_surface.cc(160):
error: more than one instance of overloaded function
"CarpetMask::isnan"
matches the argument list: function "isnan(double)" function "std::isnan(double)" argument types are:
(CCTK_REAL8)
assert(not isnan(phi)); ^compilationaborted for
/home/shamims.iiserb/ET_debug/Cactus/configs/sim/build/CarpetMask/mask_surface.cc
(code 2)make[3]: *** [mask_surface.cc.o] Error 2make[2]: ***
[make.checked]
Error 2make[1]: ***
[/home/shamims.iiserb/ET_debug/Cactus/configs/sim/lib/libthorn_CarpetMask.a]
Error 2make: *** [sim] Error 2*
I have attached the machinefile, optionlist, config log and full
output
file for reference. I configured the scripts using the available
example
optionlist and machinefile for supermuc_ng and stampede2. Please let
me
know if more info is required, and thanks in advance for your help.
Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal
ᐧ
On Fri, May 19, 2023 at 2:32 PM Shamim Haque 1910511 <
shamims@iiserb.ac.in>
wrote:
Hello Roland,
Thank you for the reply. Yes, I am trying to compile ETK in the
computing
node via sbatch queue. I am in touch with the system admin to
resolve
the issue with the compilers.
Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal
ᐧ
On Thu, May 18, 2023 at 7:32 PM Roland Haas rhaas@illinois.edu
wrote:
> Hello Shamin, > > A missing math.h would point to a faulty compiler installation.
Since
> you are trying to compile (are you?) ona compute node in a
running
job,
> my guess would be that the cluster admins are not providing
compilers
> on the compute nodes but only on the login nodes. > > This would be something to contact your cluster's help desk about. > > Yours, > Roland > > > Hello Roland, > > > > I did try that fix; the error remains the same. Actually, we
landed
on
> an > > even more primitive problem, where Steve suggested a basic
test,
that
> is to > > compile the following code: > > > > > > > > > > > > > > > > *#include <math.h>#include <stdio.h>int main() {
printf("%f\n",
> > M_PI); return 0;}* > > > > This program compiles just fine in the login node but gives out
the
> > following error while compiling it via jobscript submission in
the
> sbatch > > queue: > > > > prog.cpp:1:18: fatal error: math.h: No such file or directory > > #include <math.h> > > ^ > > compilation terminated. > > > > ----OR----- > > > > >
/home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10:
> > fatal error: features.h: No such file or directory > > 39 | #include <features.h> > > | ^~~~~~~~~~~~ > > compilation terminated. > > > > I have reported this problem to the HPC admin and await their
response.
> > > > Regards > > Shamim Haque > > Senior Research Fellow (SRF) > > Department of Physics > > IISER Bhopal > > > > ᐧ > > > > On Fri, May 12, 2023 at 12:20 AM Roland Haas <
rhaas@illinois.edu>
> wrote: > > > > > Hello all, > > > > > > Since the error is a missing M_PI and the configure script
already
> > > gives a suggested fix: > > > > > > Try adding -D_XOPEN_SOURCE to CPPFLAGS. > > > > > > have you tried adding: > > > > > > CPPFLAGS=-D_XOPEN_SOURCE > > > > > > to your option list file before compiling? > > > > > > Yours, > > > Roland > > > > > > > Thanks Steve. I have raised complaint to the admin about
this
issue.
> > > > > > > > On Thu, May 11, 2023, 11:18 PM Steven R. Brandt < > sbrandt@cct.lsu.edu> > > > wrote: > > > > > > > > > I would talk to the system administrators about that.
There
is
> > > definitely > > > > > something wrong with your compiler install if you can't
find
> math.h. > > > > > > > > > > --Steve > > > > > On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > > > > > > > > > > Dear Steve, > > > > > > > > > > I just rechecked the cpp program to output M_PI in the
computing
> nodes, > > > > > and that is not working right now, which was somehow
working
fine
> a few > > > > > days ago. I also tried different gcc versions (4.8,
10.2,
11.1,
> > > gnu8/8.3) > > > > > available in the cluster. All report similar issue: > > > > > > > > > > prog.cpp:1:18: fatal error: math.h: No such file or
directory
> > > > > #include <math.h> > > > > > ^ > > > > > compilation terminated. > > > > > > > > > > ----OR----- > > > > > > > > > > > > > >
/home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10:
> > > > > > fatal error: features.h: No such file or directory > > > > > 39 | #include <features.h> > > > > > | ^~~~~~~~~~~~ > > > > > compilation terminated. > > > > > > > > > > The cpp code seems to work fine in login node, as it
gives
out
> the > > > value > > > > > of pi upon execution. I am contacting the hpc admin to
resolve
> this > > > issue. > > > > > > > > > > I am attaching the test code (prog.cpp), the sbatch
jobscript
> (prog.sh) > > > > > and the output file (slurn-117473.out), if you would like
to
have
> a > > > look at > > > > > it. > > > > > > > > > > Please let me know if I should try something different
to
get
> this > > > working. > > > > > > > > > > Regards > > > > > Shamim Haque > > > > > Senior Research Fellow (SRF) > > > > > Department of Physics > > > > > IISER Bhopal > > > > > > > > > > ᐧ > > > > > > > > > > On Mon, May 8, 2023 at 7:25 PM Shamim Haque 1910511 < > > > shamims@iiserb.ac.in> > > > > > wrote: > > > > > > > > > >> Thanks Steve. I'll make the changes and try again. I'll
let
you
> know > > > the > > > > >> outcome. > > > > >> > > > > >> Regards > > > > >> Shamim Haque > > > > >> Senior Research Fellow (SRF) > > > > >> Department of Physics > > > > >> IISER Bhopal > > > > >> > > > > >> ᐧ > > > > >> > > > > >> On Mon, May 8, 2023 at 7:21 PM Steven R. Brandt < > sbrandt@cct.lsu.edu> > > > > >> wrote: > > > > >> > > > > >>> That tends to make me think you aren't using the
compiler
you
> think > > > > >>> you're using. You could put full paths in the .cfg
files
and
> > > recompile from > > > > >>> scratch. > > > > >>> > > > > >>> i.e. > > > > >>> > > > > >>> CXX = /full/path/to/icc > > > > >>> > > > > >>> --Steve > > > > >>> On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: > > > > >>> > > > > >>> Yes, this gives the value of Pi successfully, before
starting
> the > > > > >>> compilation: > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> */home/vkerni.iitr/sam/trial 3.141593 Current machine: > paramganga3 > > > Using > > > > >>> configuration: sim Reconfiguring sim Writing
configuration
to:
> > > > >>>
/home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList
> Cactus - > > > > >>> version: 4.12.0 Reconfiguring sim. * > > > > >>> *.* > > > > >>> *.* > > > > >>> *.* > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> *checking for M_PI... no configure: error: M_PI not
defined.
> Try > > > adding > > > > >>> -D_XOPEN_SOURCE to CPPFLAGS. Error reconfiguring
sim-config
> make: *** > > > > >>> [sim-config] Error 2 * > > > > >>> > > > > >>> > > > > >>> Shamim Haque > > > > >>> Senior Research Fellow (SRF) > > > > >>> Department of Physics > > > > >>> IISER Bhopal > > > > >>> > > > > >>> ᐧ > > > > >>> > > > > >>> On Thu, May 4, 2023 at 12:29 AM Steven Brandt < > sbrandt@cct.lsu.edu> > > > > >>> wrote: > > > > >>> > > > > >>>> So, sanity check, can you compile the following
program
> outside of > > > > >>>> Cactus? > > > > >>>> > > > > >>>> #include <math.h> > > > > >>>> #include <stdio.h> > > > > >>>> > > > > >>>> int main() { > > > > >>>> printf("%f\n", M_PI); > > > > >>>> return 0; > > > > >>>> } > > > > >>>> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: > > > > >>>> > > > > >>>> Dear Steve, > > > > >>>> > > > > >>>> I tried gcc (10 and 11) as well and I am stuck with
the
> following > > > error: > > > > >>>> *configure: error: M_PI not defined. Try adding > -D_XOPEN_SOURCE to > > > > >>>> CPPFLAGS.* > > > > >>>> > > > > >>>> This also happened for Intel (2020 and 2019). I have
attached
> both > > > the > > > > >>>> output files here. > > > > >>>> > > > > >>>> Please ignore the out file in the first email, I have > uploaded > > > another > > > > >>>> out file by mistake. The rest of the files are correct
in
the
> first > > > mail. > > > > >>>> > > > > >>>> Regards > > > > >>>> Shamim Haque > > > > >>>> Senior Research Fellow (SRF) > > > > >>>> Department of Physics > > > > >>>> IISER Bhopal > > > > >>>> > > > > >>>> ᐧ > > > > >>>> > > > > >>>> On Wed, May 3, 2023 at 8:04 PM Steven Brandt < > sbrandt@cct.lsu.edu> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> I notice from the output that your compiler claims
"limits.h"
> is > > > > >>>>> missing. Possibly there's something wrong with your
icc
> setup? > > > Maybe you > > > > >>>>> should try using gcc / gfortran instead. > > > > >>>>> > > > > >>>>> --Steve > > > > >>>>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: > > > > >>>>> > > > > >>>>> Dear Frank, > > > > >>>>> > > > > >>>>> I added the alias pattern for the computing nodes and
now
ETK
> can > > > > >>>>> detect the cluster (checked by asking "whoami")
during
the
> batch > > > queuing. > > > > >>>>> I'll get in touch with the admins to include 'patch'. > > > > >>>>> > > > > >>>>> However, the problem still persists, having assured
that
the
> > > correct > > > > >>>>> machine file and option list are being used during
the
> compilation. > > > > >>>>> > > > > >>>>> Regards > > > > >>>>> Shamim Haque > > > > >>>>> Senior Research Fellow (SRF) > > > > >>>>> Department of Physics > > > > >>>>> IISER Bhopal > > > > >>>>> > > > > >>>>> ᐧ > > > > >>>>> > > > > >>>>> On Wed, May 3, 2023 at 3:49 PM Frank Loeffler < > > > > >>>>> frank.loeffler@uni-jena.de> wrote: > > > > >>>>> > > > > >>>>>> Hi Shamim > > > > >>>>>> > > > > >>>>>> On Wed, May 03, 2023 at 03:11:49PM +0530, Shamim
Haque
> 1910511 > > > wrote: > > > > >>>>>> >We are trying to compile ETK on ParamGanga at IIT
Roorkee.
> The > > > > >>>>>> >compilation > > > > >>>>>> >stops at a very initial step and gives out the
error:
> > > > >>>>>> > > > > > >>>>>> >*checking whether the Fortran compiler (gfortran
)
> works... > > > > >>>>>> yeschecking > > > > >>>>>> >whether the Fortran compiler (gfortran ) is a > > > cross-compiler... > > > > >>>>>> nochecking > > > > >>>>>> >whether the Fortran compiler (gfortran ) supports
TYPE(*)
> for > > > > >>>>>> >CCTK_PointerTo... yeschecking how to run the C > preprocessor... > > > > >>>>>> cppchecking > > > > >>>>>> >for ANSI C header files... nochecking for C99
features...
> > > > >>>>>> yeschecking for > > > > >>>>>> >M_PI... noconfigure: error: M_PI not defined. Try
adding
> > > > >>>>>> -D_XOPEN_SOURCE to > > > > >>>>>> >CPPFLAGS.* > > > > >>>>>> > > > > > >>>>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, but
it
does
> not > > > help. > > > > >>>>>> This > > > > >>>>>> >error is consistent if we set up using Intel
Compiler
(2020
> or > > > > >>>>>> 2019), or > > > > >>>>>> >gcc (11 or 10). I have attached the outfile, error
file,
> > > machine > > > > >>>>>> script, > > > > >>>>>> >and option lists for both cases (Intel and gcc)
for
any
> > > reference. > > > > >>>>>> > > > > > >>>>>> >Secondly, we are not allowed to compile ETK on
login
nodes.
> So > > > we are > > > > >>>>>> >queuing a job script, which upon execution, enters
the
> Cactus > > > > >>>>>> directory, > > > > >>>>>> >does "setup-silent", and then executes the build
command.
> Is > > > this > > > > >>>>>> the right > > > > >>>>>> >way to compile ETK using the batch script? Or
should we
do
> > > something > > > > >>>>>> >different? I have attached the shell script
(compile.sh)
> used > > > for > > > > >>>>>> sbatch > > > > >>>>>> >queue for reference. > > > > >>>>>> > > > > >>>>>> I might be wrong, but setup-silent will not use
your
machine
> file > > > > >>>>>> which > > > > >>>>>> contains all the compiler flags, but will setup
something
> "from > > > > >>>>>> scratch" > > > > >>>>>> instead. According to the logs, this will be, e.g.,
in
> > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > >
/home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini
> > > > > >>>>>> > > > > >>>>>> My guess would be that this misses the compiler
flags
> necessary to > > > > >>>>>> have > > > > >>>>>> M_PI defined. The best way to get this working might
be
to
> let > > > > >>>>>> simfactory also detect the cluster configuration on
the
> compute > > > nodes > > > > >>>>>> (where you compile), such that setup-silent is not
needed,
> but > > > > >>>>>> instead > > > > >>>>>> the correct cluster configuration is found and used > > > automatically. In > > > > >>>>>> order to do that, look at examples of 'aliaspattern'
in
the
> > > > >>>>>> mdb/machines > > > > >>>>>> directory. An alternative would be to tell
simfactory
> specifically > > > > >>>>>> that > > > > >>>>>> you want to use your machine configuration file. > > > > >>>>>> > > > > >>>>>> As a side-note: I noticed the command 'patch' is
missing
> too. > > > This is > > > > >>>>>> a > > > > >>>>>> tool so common that it should be installed
everywhere.
Your
> admin > > > is > > > > >>>>>> probably the best person to ask for advise here. It
should
> not be > > > too > > > > >>>>>> hard to install yourself, but that should not be
necessary
> either, > > > > >>>>>> especially when you are told to compile on compute
nodes.
> > > > >>>>>> > > > > >>>>>> Frank > > > > >>>>>> > > > > >>>>>> > > > > >>>>> _______________________________________________ > > > > >>>>> Users mailing listUsers@einsteintoolkit.orghttp:// > > > lists.einsteintoolkit.org/mailman/listinfo/users > > > > >>>>> > > > > >>>>> _______________________________________________ > > > > >>>>> Users mailing list > > > > >>>>> Users@einsteintoolkit.org > > > > >>>>> > > > >
https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinf...
> > > > > >>>>> > > > > >>>> > > > > > > > > > Yours, > > > Roland > > > > > > -- > > > My email is as private as my paper mail. I therefore support > encrypting > > > and signing email messages. Get my PGP key from >
https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!5ZGhPGL_16Uj1eN8xb...
> . > > > > > -- > My email is as private as my paper mail. I therefore support
encrypting
> and signing email messages. Get my PGP key from
https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9Rc70AgC5_rUJApjF6...
.
>
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from
https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4Z697XBz_Q8JIQSrOy... .
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!74UYivLDsgDyPF-Kn9... .