<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif">Hi Roland,</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif">Thanks for your fast response!<br><br>I will just update the Einstein Toolkit.. Time flies. I didn't realise my version was now 4 years old!</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif">Chris</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 30 Sep 2020 at 09:36, Roland Haas <<a href="mailto:rhaas@illinois.edu">rhaas@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Chris,<br>
<br>
my best guess is that the MPI stack on your system got updated.<br>
<br>
MPI_UB has been removed in newer MPI standards and OpenMPI 4 does no<br>
longer offer it by default.<br>
<br>
Carpet should have stopped using it in commit<br>
5073af2caddf84b64d0aa475968cec4e9845ba73 "CarpetLib: Use<br>
MPI_Type_create_struct instead of MPI_Type_struct" from 2016.<br>
<br>
The best solution would be to update your Einstein Toolkit tree.<br>
<br>
If that is not possible you can try and see if you can downgrade or use<br>
an older MPI stack.<br>
<br>
Finally you can (if this is your workstation or so) ask Cactus to build<br>
the MPI library included in ExternalLibraries/MPI from source (which is<br>
configured to provide the old MPI_UB symbol) by setting:<br>
<br>
MPI_DIR=BUILD <br>
<br>
in your option list and compiling from scratch (make sim-realclean).<br>
<br>
Yours,<br>
Roland<br>
<br>
> Hi everyone,<br>
> <br>
> I've managed to get this compiling and have just completed the example BBH<br>
> run simulating the system of the first detection, however now that I have<br>
> left it for a few weeks and come back, naturally nothing works.<br>
> <br>
> The error I now have is with MPI_UB and dh.cc when building through<br>
> simfactory. I get the following output:<br>
> <br>
> In file included from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dist.hh:12,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/vect.hh:13,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/bbox.hh:11,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:10:<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc: In<br>
> function ‘ompi_datatype_t* mpi_datatype(const dh::light_dboxes&)’:<br>
> /opt/openmpi4/include/mpi.h:337:63: error: expected ‘}’ before ‘;’ token<br>
> #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(symbol, new_symbol) 0;<br>
> static_assert(0, #symbol " was removed in MPI-3.0. Use " #new_symbol "<br>
> instead.")<br>
> ^<br>
> /opt/openmpi4/include/mpi.h:1131:24: note: in expansion of macro<br>
> ‘THIS_SYMBOL_WAS_REMOVED_IN_MPI30’<br>
> # define MPI_UB THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_UB,<br>
> MPI_Type_create_resized);<br>
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:23:<br>
> note: in expansion of macro ‘MPI_UB’<br>
> {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}};<br>
> ^~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:9:<br>
> note: to match this ‘{’<br>
> {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}};<br>
> ^<br>
> In file included from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dist.hh:12,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/vect.hh:13,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/bbox.hh:11,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:10:<br>
> /opt/openmpi4/include/mpi.h:337:63: error: expected ‘}’ before ‘;’ token<br>
> #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(symbol, new_symbol) 0;<br>
> static_assert(0, #symbol " was removed in MPI-3.0. Use " #new_symbol "<br>
> instead.")<br>
> ^<br>
> /opt/openmpi4/include/mpi.h:1131:24: note: in expansion of macro<br>
> ‘THIS_SYMBOL_WAS_REMOVED_IN_MPI30’<br>
> # define MPI_UB THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_UB,<br>
> MPI_Type_create_resized);<br>
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:23:<br>
> note: in expansion of macro ‘MPI_UB’<br>
> {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}};<br>
> ^~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2188:46:<br>
> note: to match this ‘{’<br>
> dist::mpi_struct_descr_t const descr[] = {<br>
> ^<br>
> In file included from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dist.hh:12,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/vect.hh:13,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/bbox.hh:11,<br>
> from<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:10:<br>
> /opt/openmpi4/include/mpi.h:337:79: error: static assertion failed: MPI_UB<br>
> was removed in MPI-3.0. Use MPI_Type_create_resized instead.<br>
> #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(symbol, new_symbol) 0;<br>
> static_assert(0, #symbol " was removed in MPI-3.0. Use " #new_symbol "<br>
> instead.")<br>
> <br>
> ^<br>
> /opt/openmpi4/include/mpi.h:1131:24: note: in expansion of macro<br>
> ‘THIS_SYMBOL_WAS_REMOVED_IN_MPI30’<br>
> # define MPI_UB THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_UB,<br>
> MPI_Type_create_resized);<br>
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:23:<br>
> note: in expansion of macro ‘MPI_UB’<br>
> {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}};<br>
> ^~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:29:<br>
> error: expected primary-expression before ‘,’ token<br>
> {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}};<br>
> ^<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2188:36:<br>
> warning: unused variable ‘descr’ [-Wunused-variable]<br>
> dist::mpi_struct_descr_t const descr[] = {<br>
> ^~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2177:23:<br>
> warning: unused variable ‘newtype’ [-Wunused-variable]<br>
> static MPI_Datatype newtype;<br>
> ^~~~~~~<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:50:<br>
> warning: no return statement in function returning non-void [-Wreturn-type]<br>
> {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}};<br>
> ^<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc: At<br>
> global scope:<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2197:5:<br>
> error: ‘newtype’ does not name a type; did you mean ‘iswctype’?<br>
> newtype = dist::create_mpi_datatype(sizeof descr / sizeof descr[0],<br>
> descr,<br>
> ^~~~~~~<br>
> iswctype<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2208:5:<br>
> error: ‘initialised’ does not name a type; did you mean ‘MPI_Initialized’?<br>
> initialised = true;<br>
> ^~~~~~~~~~~<br>
> MPI_Initialized<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2209:3:<br>
> error: expected declaration before ‘}’ token<br>
> }<br>
> ^<br>
> /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2177:23:<br>
> warning: ‘newtype’ defined but not used [-Wunused-variable]<br>
> static MPI_Datatype newtype;<br>
> ^~~~~~~<br>
> make[3]: ***<br>
> [/maybehome/cstevens/Cactus/configs/CTGD_Testing/config-data/make.config.rules:281:<br>
> dh.cc.o] Error 1<br>
> make[2]: *** [/maybehome/cstevens/Cactus/lib/make/make.thornlib:113:<br>
> make.checked] Error 2<br>
> make[1]: *** [/maybehome/cstevens/Cactus/lib/make/make.configuration:181:<br>
> /maybehome/cstevens/Cactus/configs/CTGD_Testing/lib/libthorn_CarpetLib.a]<br>
> Error 2<br>
> make: *** [Makefile:260: CTGD_Testing] Error 2<br>
> <br>
> I did find the following thread:<br>
> <br>
> <a href="http://lists.einsteintoolkit.org/pipermail/users/2018-October/006566.html" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/pipermail/users/2018-October/006566.html</a><br>
> <br>
> and I've done a clean compilation but same deal. I'm using OpenMPI 4.0.5<br>
> and gcc/g++ etc 8.3.1. I've confirmed which mpicc etc points to the right<br>
> place, and what I think the relevant lines in the option list file are:<br>
> <br>
> CPPFLAGS = -DOMPI_SKIP_MPICXX<br>
> FPPFLAGS = -traditional<br>
> CFLAGS = -g3 -std=gnu11 -Wunused-variable<br>
> CXXFLAGS = -g3 -std=gnu++11 -Wunused-variable<br>
> F77FLAGS = -g3 -fcray-pointer -ffixed-line-length-none<br>
> F90FLAGS = -g3 -fcray-pointer -ffixed-line-length-none# -Wextra<br>
> -Wimplicit-interface#-fallow-argument-mismatch<br>
> <br>
> MPI_DIR = /opt/openmpi4/<br>
> MPI_LIB_DIRS = /opt/openmpi4/lib<br>
> MPI_INC_DIRS = /opt/openmpi4/include<br>
> MPI_LIBS = mpi open-rte open-pal<br>
> <br>
> I have added CPP = ... CXX = ... etc which point to the appropriate places.<br>
> <br>
> I also have<br>
> <br>
> cstevens@skynet:~/Cactus$ mpic++ --showme<br>
> g++ -I/opt/openmpi4/include -pthread -Wl,-rpath -Wl,/opt/openmpi4/lib<br>
> -Wl,--enable-new-dtags -L/opt/openmpi4/lib -lmpi<br>
> <br>
> Any help would be greatly appreciated.<br>
> <br>
> Thanks,<br>
> <br>
> Chris<br>
> <br>
<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" rel="noreferrer" target="_blank">http://pgp.mit.edu</a> .<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><pre cols="72">Dr Chris Stevens<br>Postdoctoral Fellow<br>Rm 310A<br>Department of Mathematics and Statistics<br>University of Otago<br>Ph: +64 3 479 4567<br><br><a href="http://www.chrisdoesmaths.com" target="_blank">http://www.chrisdoesmaths.com</a><br><a href="http://gravity.otago.ac.nz" target="_blank">http://gravity.otago.ac.nz</a><br></pre>
</div></div></div></div></div></div></div></div></div></div></div></div></div></div></span></div></div>