<div dir="ltr"><div class="gmail_default" style="color:#000000">Hi Roland,</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">Thanks for the help. Turns out libmpi_cxx.so was not present in this particular openmpi library. So I compiled with another version of openmpi which has this file, and now ET seems to be working fine after a successful compilation. Meanwhile, I am working out this issue with cluster admins to resolve the issue issue with openmpi-x86_64.</div><div class="gmail_default" style="color:#000000"><br></div><div class="gmail_default" style="color:#000000">Regards</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font color="#666666">Shamim Haque</font></div><div dir="ltr"><font color="#666666">Senior Research Fellow (SRF)<br></font><div><font color="#666666">Department of Physics</font></div><div><font color="#666666">IISER Bhopal</font></div></div></div></div></div></div></div></div></div></div><br></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=ac2hhbWltc0BpaXNlcmIuYWMuaW4%3D&type=zerocontent&guid=4db1f53a-162d-43d7-85e7-3d60198a22b5"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 16, 2021 at 9:20 PM Roland Haas <<a href="mailto:rhaas@illinois.edu">rhaas@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Shamim Haque,<br>
<br>
> I have done it successfully once with OpenMPI-x86_64 in our cluster, but it<br>
> turns out this MPI is only available in the login node and not available on<br>
> computing nodes, hence, I am not able to use it once a job is submitted in<br>
> the queue.<br>
This is something you should bring up with the cluster admins. Having<br>
MPI only on the login nodes but not on the compute nodes makes no sense<br>
since MPI is an inter-node communication library.<br>
<br>
> I could not figure out the issue here. It seems<br>
> */usr/include/openmpi-x86_64/openmpi/ompi/mpi<br>
> *is being taken by ET from somewhere and it is not taking OpenMPI 2.1.0.<br>
My guess based on the error message being about MPI C++ bindings is<br>
that the issue is similar to this recent ET ticket:<br>
<br>
<a href="https://bitbucket.org/einsteintoolkit/tickets/issues/2578/kranc-cactus-and-mpi-linking-issues" rel="noreferrer" target="_blank">https://bitbucket.org/einsteintoolkit/tickets/issues/2578/kranc-cactus-and-mpi-linking-issues</a><br>
<br>
namely you have to manually set MPI_LIBS in you option list to include<br>
mpi_cxx (or the equivalent name for your cluster).<br>
<br>
You can find out the other libraries to include by quetying mpic++:<br>
<br>
mpic++ -showme:link<br>
<br>
which will output the <br>
<br>
MPI_LIB_DIRS <br>
<br>
setting (the directory after -L) and the libraries afterwards (you need<br>
to remove the -l to get the actual name) and use<br>
<br>
mpic++ -showme:compile<br>
<br>
to similarly get the directory to put into the <br>
<br>
MPI_INC_DIRS<br>
<br>
variable. You should then set MPI_DIR to the directory containing<br>
bin/mpic++<br>
<br>
Ie if "which mpic++" returns /usr/local/openmpi/bin/mpic++ you would<br>
use /usr/local/openmpi<br>
<br>
Note that OpenMPI 2.1 is somewhat old (from 2019) and no longer among<br>
the supported releases by the OpenMPI team, so you may also ask the<br>
admins if there is any newer version available.<br>
<br>
Yours,<br>
Roland<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
</blockquote></div>