Hi Roland,
Thanks for your fast response!
I will just update the Einstein Toolkit.. Time flies. I didn't realise my version was now 4 years old!
Thanks,
Chris
On Wed, 30 Sep 2020 at 09:36, Roland Haas rhaas@illinois.edu wrote:
Hello Chris,
my best guess is that the MPI stack on your system got updated.
MPI_UB has been removed in newer MPI standards and OpenMPI 4 does no longer offer it by default.
Carpet should have stopped using it in commit 5073af2caddf84b64d0aa475968cec4e9845ba73 "CarpetLib: Use MPI_Type_create_struct instead of MPI_Type_struct" from 2016.
The best solution would be to update your Einstein Toolkit tree.
If that is not possible you can try and see if you can downgrade or use an older MPI stack.
Finally you can (if this is your workstation or so) ask Cactus to build the MPI library included in ExternalLibraries/MPI from source (which is configured to provide the old MPI_UB symbol) by setting:
MPI_DIR=BUILD
in your option list and compiling from scratch (make sim-realclean).
Yours, Roland
Hi everyone,
I've managed to get this compiling and have just completed the example
BBH
run simulating the system of the first detection, however now that I have left it for a few weeks and come back, naturally nothing works.
The error I now have is with MPI_UB and dh.cc when building through simfactory. I get the following output:
In file included from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dist.hh:12, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/vect.hh:13, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/bbox.hh:11, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:10: /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc: In function ‘ompi_datatype_t* mpi_datatype(const dh::light_dboxes&)’: /opt/openmpi4/include/mpi.h:337:63: error: expected ‘}’ before ‘;’ token #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(symbol, new_symbol) 0; static_assert(0, #symbol " was removed in MPI-3.0. Use " #new_symbol " instead.") ^ /opt/openmpi4/include/mpi.h:1131:24: note: in expansion of macro ‘THIS_SYMBOL_WAS_REMOVED_IN_MPI30’ # define MPI_UB THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_UB, MPI_Type_create_resized); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:23:
note: in expansion of macro ‘MPI_UB’ {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}}; ^~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:9:
note: to match this ‘{’ {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}}; ^ In file included from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dist.hh:12, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/vect.hh:13, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/bbox.hh:11, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:10: /opt/openmpi4/include/mpi.h:337:63: error: expected ‘}’ before ‘;’ token #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(symbol, new_symbol) 0; static_assert(0, #symbol " was removed in MPI-3.0. Use " #new_symbol " instead.") ^ /opt/openmpi4/include/mpi.h:1131:24: note: in expansion of macro ‘THIS_SYMBOL_WAS_REMOVED_IN_MPI30’ # define MPI_UB THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_UB, MPI_Type_create_resized); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:23:
note: in expansion of macro ‘MPI_UB’ {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}}; ^~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2188:46:
note: to match this ‘{’ dist::mpi_struct_descr_t const descr[] = { ^ In file included from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dist.hh:12, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/vect.hh:13, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/bbox.hh:11, from /maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:10: /opt/openmpi4/include/mpi.h:337:79: error: static assertion failed:
MPI_UB
was removed in MPI-3.0. Use MPI_Type_create_resized instead. #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(symbol, new_symbol) 0; static_assert(0, #symbol " was removed in MPI-3.0. Use " #new_symbol " instead.")
^ /opt/openmpi4/include/mpi.h:1131:24: note: in expansion of macro ‘THIS_SYMBOL_WAS_REMOVED_IN_MPI30’ # define MPI_UB THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_UB, MPI_Type_create_resized); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:23:
note: in expansion of macro ‘MPI_UB’ {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}}; ^~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:29:
error: expected primary-expression before ‘,’ token {1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}}; ^
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2188:36:
warning: unused variable ‘descr’ [-Wunused-variable] dist::mpi_struct_descr_t const descr[] = { ^~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2177:23:
warning: unused variable ‘newtype’ [-Wunused-variable] static MPI_Datatype newtype; ^~~~~~~
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2195:50:
warning: no return statement in function returning non-void
[-Wreturn-type]
{1, sizeof s, MPI_UB, "MPI_UB", "MPI_UB"}}; ^/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc: At global scope:
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2197:5:
error: ‘newtype’ does not name a type; did you mean ‘iswctype’? newtype = dist::create_mpi_datatype(sizeof descr / sizeof descr[0], descr, ^~~~~~~ iswctype
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2208:5:
error: ‘initialised’ does not name a type; did you mean
‘MPI_Initialized’?
initialised = true; ^~~~~~~~~~~ MPI_Initialized/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2209:3:
error: expected declaration before ‘}’ token } ^
/maybehome/cstevens/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2177:23:
warning: ‘newtype’ defined but not used [-Wunused-variable] static MPI_Datatype newtype; ^~~~~~~ make[3]: ***
[/maybehome/cstevens/Cactus/configs/CTGD_Testing/config-data/make.config.rules:281:
dh.cc.o] Error 1 make[2]: *** [/maybehome/cstevens/Cactus/lib/make/make.thornlib:113: make.checked] Error 2 make[1]: *** [/maybehome/cstevens/Cactus/lib/make/make.configuration:181: /maybehome/cstevens/Cactus/configs/CTGD_Testing/lib/libthorn_CarpetLib.a] Error 2 make: *** [Makefile:260: CTGD_Testing] Error 2
I did find the following thread:
http://lists.einsteintoolkit.org/pipermail/users/2018-October/006566.html
and I've done a clean compilation but same deal. I'm using OpenMPI 4.0.5 and gcc/g++ etc 8.3.1. I've confirmed which mpicc etc points to the right place, and what I think the relevant lines in the option list file are:
CPPFLAGS = -DOMPI_SKIP_MPICXX FPPFLAGS = -traditional CFLAGS = -g3 -std=gnu11 -Wunused-variable CXXFLAGS = -g3 -std=gnu++11 -Wunused-variable F77FLAGS = -g3 -fcray-pointer -ffixed-line-length-none F90FLAGS = -g3 -fcray-pointer -ffixed-line-length-none# -Wextra -Wimplicit-interface#-fallow-argument-mismatch
MPI_DIR = /opt/openmpi4/ MPI_LIB_DIRS = /opt/openmpi4/lib MPI_INC_DIRS = /opt/openmpi4/include MPI_LIBS = mpi open-rte open-pal
I have added CPP = ... CXX = ... etc which point to the appropriate
places.
I also have
cstevens@skynet:~/Cactus$ mpic++ --showme g++ -I/opt/openmpi4/include -pthread -Wl,-rpath -Wl,/opt/openmpi4/lib -Wl,--enable-new-dtags -L/opt/openmpi4/lib -lmpi
Any help would be greatly appreciated.
Thanks,
Chris
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .