Hello all,
I am trying to install ET with OpenMPI 2.1.0 but it seems to be causing issues.
I have done it successfully once with OpenMPI-x86_64 in our cluster, but it turns out this MPI is only available in the login node and not available on computing nodes, hence, I am not able to use it once a job is submitted in the queue.
I could find OpenMPI 2.1.0 being available on computing nodes, so I was trying to compile with this version. I created a module file to load this module before installation. I could see the following in the terminal, which I guess, ET could recognise OpenMPI 2.1.0:
Running configuration script for thorn MPI: MPI selected, but MPI_DIR is not set. Computing settings... Found MPI compiler wrapper at /opt/mpi/openmpi/2.1.0/bin/mpic++! Successfully configured MPI. Finished running configuration script for thorn MPI.
Later, the progress stops with the following error:
/home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o): In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/op_inln.h:121: undefined reference to `ompi_mpi_cxx_op_intercept' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o): In function `Intracomm': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o): In function `Intracomm': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined reference to `MPI::Comm::Comm()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o): In function `Cartcomm': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined reference to `MPI::Comm::Comm()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o): In function `Graphcomm': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined reference to `MPI::Comm::Comm()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o): In function `Intracomm': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined reference to `MPI::Comm::Comm()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o):/usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm.h:25: more undefined references to `MPI::Comm::Comm()' follow /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o):(.rodata._ZTVN3MPI8DatatypeE[_ZTVN3MPI8DatatypeE]+0x78): undefined reference to `MPI::Datatype::Free()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CT_MultiLevel.a(CT_MultiLevel.cc.o):(.rodata._ZTVN3MPI3WinE[_ZTVN3MPI3WinE]+0x48): undefined reference to `MPI::Win::Free()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CarpetLib.a(backtrace.cc.o): In function `MPI::Intracomm::Intracomm()': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined reference to `MPI::Comm::Comm()' /home2/mallick/ET3/Cactus/configs/sim/lib/libthorn_CarpetLib.a(backtrace.cc.o): In function `MPI::Intracomm::Intracomm(ompi_communicator_t*)': /usr/include/openmpi-x86_64/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()' collect2: error: ld returned 1 exit status make[1]: *** [/home2/mallick/ET3/Cactus/exe/cactus_sim] Error 1 make: *** [sim] Error 2
I could not figure out the issue here. It seems */usr/include/openmpi-x86_64/openmpi/ompi/mpi *is being taken by ET from somewhere and it is not taking OpenMPI 2.1.0.
Some suggestions would be very helpful. Thank you.
Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal
Hello Shamim Haque,
I have done it successfully once with OpenMPI-x86_64 in our cluster, but it turns out this MPI is only available in the login node and not available on computing nodes, hence, I am not able to use it once a job is submitted in the queue.
This is something you should bring up with the cluster admins. Having MPI only on the login nodes but not on the compute nodes makes no sense since MPI is an inter-node communication library.
I could not figure out the issue here. It seems */usr/include/openmpi-x86_64/openmpi/ompi/mpi *is being taken by ET from somewhere and it is not taking OpenMPI 2.1.0.
My guess based on the error message being about MPI C++ bindings is that the issue is similar to this recent ET ticket:
https://bitbucket.org/einsteintoolkit/tickets/issues/2578/kranc-cactus-and-m...
namely you have to manually set MPI_LIBS in you option list to include mpi_cxx (or the equivalent name for your cluster).
You can find out the other libraries to include by quetying mpic++:
mpic++ -showme:link
which will output the
MPI_LIB_DIRS
setting (the directory after -L) and the libraries afterwards (you need to remove the -l to get the actual name) and use
mpic++ -showme:compile
to similarly get the directory to put into the
MPI_INC_DIRS
variable. You should then set MPI_DIR to the directory containing bin/mpic++
Ie if "which mpic++" returns /usr/local/openmpi/bin/mpic++ you would use /usr/local/openmpi
Note that OpenMPI 2.1 is somewhat old (from 2019) and no longer among the supported releases by the OpenMPI team, so you may also ask the admins if there is any newer version available.
Yours, Roland
Hi Roland,
Thanks for the help. Turns out libmpi_cxx.so was not present in this particular openmpi library. So I compiled with another version of openmpi which has this file, and now ET seems to be working fine after a successful compilation. Meanwhile, I am working out this issue with cluster admins to resolve the issue issue with openmpi-x86_64.
Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal
ᐧ
On Thu, Dec 16, 2021 at 9:20 PM Roland Haas rhaas@illinois.edu wrote:
Hello Shamim Haque,
I have done it successfully once with OpenMPI-x86_64 in our cluster, but
it
turns out this MPI is only available in the login node and not available
on
computing nodes, hence, I am not able to use it once a job is submitted
in
the queue.
This is something you should bring up with the cluster admins. Having MPI only on the login nodes but not on the compute nodes makes no sense since MPI is an inter-node communication library.
I could not figure out the issue here. It seems */usr/include/openmpi-x86_64/openmpi/ompi/mpi *is being taken by ET from somewhere and it is not taking OpenMPI 2.1.0.
My guess based on the error message being about MPI C++ bindings is that the issue is similar to this recent ET ticket:
https://bitbucket.org/einsteintoolkit/tickets/issues/2578/kranc-cactus-and-m...
namely you have to manually set MPI_LIBS in you option list to include mpi_cxx (or the equivalent name for your cluster).
You can find out the other libraries to include by quetying mpic++:
mpic++ -showme:link
which will output the
MPI_LIB_DIRS
setting (the directory after -L) and the libraries afterwards (you need to remove the -l to get the actual name) and use
mpic++ -showme:compile
to similarly get the directory to put into the
MPI_INC_DIRS
variable. You should then set MPI_DIR to the directory containing bin/mpic++
Ie if "which mpic++" returns /usr/local/openmpi/bin/mpic++ you would use /usr/local/openmpi
Note that OpenMPI 2.1 is somewhat old (from 2019) and no longer among the supported releases by the OpenMPI team, so you may also ask the admins if there is any newer version available.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .
users@lists.einsteintoolkit.org
-
Roland Haas -
Shamim Haque