Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

Carpet documentation?
by Jonah Miller 30 Mar '18

30 Mar '18

Hello all, I seem to recall that there was documentation for setting up refinement in Carpet---i.e., what parameters like Shiftout and centering do the grid setup. I thought it was one of the ThornDoc files for Carpet, but I can no longer find it. Does anyone know where it might be? Thanks! Best, Jonah Miller

2 2

Generalising to OpenMP
by Chris Stevens 29 Mar '18

29 Mar '18

Hi everyone, I am new to OpenMP programming, and have quickly come across an (I think) interesting task that I want to parallelise with OpenMP that does not seem straightforwardly generalisable. I would like to hear your feedback on the potential solutions and whether there is a better way to do it: Task: Create a scratch array, say of CCTK_REAL, during an initialisation phase that is shared with other code using a header file. This array is defined only once. This scratch array is then used as temporary storage space in a function that does some computation within a for loop. i.e. function blah () { for (....) { computation involving scratch array } } This works fine when OpenMP is turned off. However when you add to the for loop above #pragma omp parallel for, then one sees that this one scratch array will be being used by multiple threads at once, thus causing issues. Solution 1: Each thread has its own scratch space. I am not sure if it is within the OpenMP paradigm to create scratch spaces for each thread within the initialisation file, and share them through a header. This doesn't seem to work and I am thinking this shouldn't be possible? Solution 2: Create one serial scratch space during initialisation that has size dependent on omp_get_num_threads(). One would then have to change the computation involving the scratch array to be dependent on omp_get_thread_num() so as to use its allocated part of the scratch space. Solution 3: One could do something like #pragma omp parallel { create scratch space #pragma omp parallel for for { computation involving thread specific scratch space } } In summary: Solution 1 doesn't seem to work and probably doesn't fit into the OpenMP paradigm. Solution 2 would work but doesn't seem very nice. Solution 3 is nicer but inefficient as I don't want to be creating scratch spaces all the time. Is there perhaps another way that fits better with the OpenMP paradigm? Many thanks! Chris -- Dr Chris Stevens Claude Leon Postdoctoral Fellow Department of Mathematics Rhodes University Room 5 Ph: +27 46 603 8932 Web: www.chrisdoesmaths.com <http://www.chrisdoesmaths.com>

2 3

Error in output of helloworld
by Swarnim Shashank 27 Mar '18

27 Mar '18

Hello, I am an undergraduate student and I am learning ET for my project. I am having trouble running the helloworld simulation. Everything works out but when I do "show-output" I get this on my terminal: Simulation name: helloworld Warning: job status is U Warning: job status is U Warning: Job is not running, cannot retreive exechost ================================================================================ The job's Formaline output is: ================================================================================ (file does not exist) ================================================================================ The job's stdout is: ================================================================================ Simulation name: helloworld Running simulation helloworld Preparing: Checking: /home/swarnim/simulations/helloworld/output-0000-active koszmikroid-X550LD Mon Mar 26 17:44:46 IST 2018 Environment: Starting: INFO (Cactus): Increasing logging level from 0 to 3 Mon Mar 26 17:44:47 IST 2018 Simfactory Done at date: 0 ================================================================================ The job's stderr is: ================================================================================ + set -e + cd /home/swarnim/simulations/helloworld/output-0000-active + echo Checking: + pwd + hostname + date + echo Environment: + export CACTUS_NUM_PROCS=1 + export CACTUS_NUM_THREADS=2 + export GMON_OUT_PREFIX=gmon.out + export OMP_NUM_THREADS=2 + env + sort + echo Starting: + date +%s + export CACTUS_STARTTIME=1522066486 + [ 1 = 1 ] + [ 0 -eq 0 ] + /home/swarnim/simulations/helloworld/SIMFACTORY/exe/cactus_sim -L 3 /home/swarnim/simulations/helloworld/output-0000/HelloWorld.par Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: PMPI_Comm_rank(110): MPI_Comm_rank(comm=0xe0cc28e0, rank=0x7ffd2aa27d38) failed PMPI_Comm_rank(68).: Invalid communicator [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=873024773 : system msg for write_line failure : Bad file descriptor ================================================================================ Please let me know what the problem is and how to sort it. Thank you Sincerely *Swarnim Shashank* Fourth Year Integrated MSc. Student UM-DAE-Centre for Excellence in Basic Sciences Mumbai

2 1

meeting minutes for 2018-03-26
by Roland Haas 26 Mar '18

26 Mar '18

Present: Roland, Helvi, Peter, Sam, Steve, Bhavesh, Zach, Antoni, Qian Einstein Toolkit Maintainers: * list is outdated: http://einsteintoolkit.org/maintainers-credits.html ** add Steve Brandt ** update status of persons SSL status of ET websites: * apache config files seem to indicate that the correct certs should be in use but the monitor shows otherwise * Steve is working on this with LSU computer support Suggest to move to bitbucket: * would relieve burden from Steve to be sole website maintainer * would require ssl ready redirect from https;//www.einsteintoolkit.org since we cannot directly point the DNS entry to the bitbucket site (cert would not match) * User and Reference documentation do not work yet * please poke around in website to find outdated information Performance working group: * try to set up interaction with scheduling working group Suggestion: * evaluate need / work required for a full discontinuous Galerkin method driver with HP refinement ReCAPTCHA issue: * currently no captchas are required to edit the page * Steve is working on fixing it before the deadline Wiki page: * add direct link to landing page of the ET: approved US ET meeting: * in process of securing hosting at GT, Bhavesh will send emails once hosting is sorted out Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

performance optimization working group
by Roland Haas 25 Mar '18

25 Mar '18

Dear all, after the discussions in the Einstein Toolkit workshop at NCSA we have formed a new working group "Performance optimization" led by Erik and myself. This group will serve as an arena to coordinate activities to improve the Einstein Toolkit performance through code optimization, algorithmic improvements and code generation to enable new and exciting physical applications in the Einstein Toolkit. You can find a more detailed description of the working group here https://docs.einsteintoolkit.org/et-docs/Working_Group_on_Performance_Optim… If you are interested in joining the working group please sign up for our mailing list performanceoptimization-wg(a)einsteintoolkit.org here http://lists.einsteintoolkit.org/mailman/listinfo/performanceoptimization-wg The working group document will be presented to the ET maintainers in the ET call on Monday March 26 2018 and if all goes well will be approved a week after that. Best wishes, Roland, Erik, Zach -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Re: [Users] [External - Warning: Suspect Attachment] null Re: mpi.h 9%
by Roland Haas 25 Mar '18

25 Mar '18

Hello DG, it looks like you are trying to use the system mpicc to compile OpenMPI in Cactus' ExternalLibraries. This is a very odd thing to do and I am not surprised that you would fail attempting this. Instead why not set MPI_DIR = <PATH-TO-DIRECTORY-WHERE-MPICC-IC>/.. ie the parent directory of the system MPI's "bin" directory? Leave the other MPI settings blank and Cactus should obtain the required settings from the system MPI that it will find in MPI_DIR. Yours, Roland > Hello, > > Using mpicc, etc… seems to work a lot better but I’m still having a couple issues. When I use the cluster’s mpi, I get the error > > ld: cannot find -lmpi_usempif08 > make[2]: *** [/home/dgarriso/Cactus/exe/cactus_fixedmhd] Error 1 > make[1]: *** [fixedmhd] Error 2 > make: *** [fixedmhd-rebuild] Error 2 > > When I try to build the Cactus mpi, it fails > > Making all in libevent > Making all in . > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT event.lo -MD -MP -MF .deps/event.Tpo -c event.c -o event.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evthread.lo -MD -MP -MF .deps/evthread.Tpo -c evthread.c -o evthread.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evmap.lo -MD -MP -MF .deps/evmap.Tpo -c evmap.c -o evmap.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT log.lo -MD -MP -MF .deps/log.Tpo -c log.c -o log.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evutil.lo -MD -MP -MF .deps/evutil.Tpo -c evutil.c -o evutil.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evutil_rand.lo -MD -MP -MF .deps/evutil_rand.Tpo -c evutil_rand.c -o evutil_rand.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT strlcpy.lo -MD -MP -MF .deps/strlcpy.Tpo -c strlcpy.c -o strlcpy.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT select.lo -MD -MP -MF .deps/select.Tpo -c select.c -o select.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT poll.lo -MD -MP -MF .deps/poll.Tpo -c poll.c -o poll.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT epoll.lo -MD -MP -MF .deps/epoll.Tpo -c epoll.c -o epoll.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT signal.lo -MD -MP -MF .deps/signal.Tpo -c signal.c -o signal.o > signal.c(135): error: identifier "NSIG" is undefined > int ncaught[NSIG]; > ^ > > signal.c(253): error: identifier "SA_RESTART" is undefined > sa.sa_flags |= SA_RESTART; > ^ > > signal.c(281): error: identifier "NSIG" is undefined > EVUTIL_ASSERT(evsignal >= 0 && evsignal < NSIG); > ^ > > signal.c(355): error: identifier "NSIG" is undefined > EVUTIL_ASSERT(evsignal >= 0 && evsignal < NSIG); > ^ > > signal.c(409): error: identifier "NSIG" is undefined > for (i = 0; i < NSIG; ++i) { > ^ > > compilation aborted for signal.c (code 2) > make[10]: *** [signal.lo] Error 1 > make[9]: *** [all-recursive] Error 1 > make[8]: *** [all] Error 2 > make[7]: *** [all-recursive] Error 1 > make[6]: *** [all-recursive] Error 1 > make[5]: *** [all-recursive] Error 1 > Died at /home/dgarriso/Cactus/arrangements/ExternalLibraries/MPI/src/build.pl line 74. > make[4]: *** [/home/dgarriso/Cactus/configs/fixedmhd/scratch/done/MPI] Error 17 > make[3]: *** [make.checked] Error 2 > make[2]: *** [/home/dgarriso/Cactus/configs/fixedmhd/lib/libthorn_MPI.a] Error 2 > make[1]: *** [fixedmhd] Error 2 > make: *** [fixedmhd-rebuild] Error 2 > > Is this an issue with the cluster mpi installation or with how cactus works with the compilers? I am using Intel compilers from 2015 (intel/ics-2015-64-bit), and openmpi version 1.6.5 on the cluster compiled with intel compilers. > > -DG > > > On Mar 21, 2018, at 9:52 AM, Roland Haas <rhaas(a)illinois.edu> wrote: > > > > Hello DG, > > > > hmm that all look fine to me. From the content of > > configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it is using a system installed MPI (openmpi-1.6.5 in /share/apps) which makes sense given that you pass MPI_DIR=/share/apps/openmpi-1.6.5 in your config-info (which is basically a copy of your option list). So MPI_DIR=BUILD was not used in the end (through an oversight or something else I guess). > > > > Since you are passing an explicit > > > > MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > > > > I would definitely myself try and see if indeed there is a file > > > > /share/apps/openmpi-1.6.5/include/mpi.h > > > > and that this file is readable to your user account (eg does > > less /share/apps/openmpi-1.6.5/include/mpi.h work?). > > > > This is just in case /share/apps/openmpi-1.6.5 was eg retired by the > > sysadmins. > > > > Yours, > > Roland > > > >> Hello, > >> > >> Please see below. I’m using UH’s Maxwell cluster and don’t use simfactory. > >> > >> Thanks, > >> > >> -DG > >> > >>> On Mar 20, 2018, at 11:47 AM, Roland Haas <roland.haas(a)physics.gatech.edu> wrote: > >>> > >>> Hello DG, > >>> > >>> it could be that something failed while MPI was compiled. Would you mind attaching the following files from inside of your Cactus tree, please? > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.MPI.defn > >> > >> INC_DIRS += $(MPI_INC_DIRS) > >> INC_DIRS_F += $(MPI_LIB_DIRS) > >> MPI_BUILD = > >> MPI_INSTALL_DIR = > >> HWLOC_DIR = > >> CCTK_MPI = 1 > >> MPI_DIR = /share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include > >> MPI_LIB_DIRS = /share/apps/openmpi-1.6.5/lib > >> MPI_LIBS = mpi mpi_cxx open-rte open-pal dl nsl util > >> HAVE_CAPABILITY_MPI = 1 > >> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.defn > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.defn > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.defn > >> INC_DIRS += $(HDF5_INC_DIRS) > >> INC_DIRS_F += $(HDF5_INC_DIRS) > >> HDF5_BUILD = > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> LIBSZ_DIR = > >> LIBZ_DIR = > >> HDF5_INSTALL_DIR = > >> HDF5_DIR = /share/apps/hdf5-1.8.12 > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> HDF5_INC_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/hdf5-1.8.12/include /share/apps/openmpi-1.6.5/include > >> HDF5_LIB_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/openmpi-1.6.5/lib > >> HDF5_LIBS = hdf5hl_fortran hdf5_fortran hdf5_hl hdf5 z mpi mpi_cxx open-rte open-pal dl nsl util m z > >> HAVE_CAPABILITY_HDF5 = 1 > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.deps > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.deps > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.deps > >> > >>> > >>> as well as > >>> > >>> confis/sim/config-info (as well as the option list that you used)? > >> > >> # CONFIGURATION : fixedmhd > >> # CONFIG-DATE : Mon Mar 19 01:07:33 2018 (GMT) > >> # CONFIG-HOST : cusco.hpcc.uh.edu > >> # CONFIG-STATUS : 0 > >> # CONFIG-OPTIONS : > >> CC=icc > >> CPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> CXX=icpc > >> CXX_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> C_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> F77=ifort > >> F77_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> F90=ifort > >> F90_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> FFLAGS=-O3 -fast -Mipa=fast,inline -tp=x64 > >> FFTW=yes > >> FFTW_DIR=/home/dgarriso/fftw3 > >> FFTW_LIBS=fftw3_threads fftw3_mpi $(MPI_LIBS) fftw3 m > >> FPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> HDF5_DIR=/share/apps/hdf5-1.8.12 > >> LAPACK=yes > >> LAPACK_DIR=/home/dgarriso/ifort64/lib > >> LAPACK_LIBS=acml ifcoremt_pic imf irc svml > >> MPI_DIR=/share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > >> MPI_LIBS=mpi mpi_cxx open-rte open-pal dl nsl util > >> MPI_LIB_DIRS=/share/apps/openmpi-1.6.5/lib > >> OPTIMISE=yes > >> PTHREADS=no > >> VERBOSE=yes > >> > >>> > >>> If possible also the full output of: > >>> > >>> export VERBOSE=yes > >>> simfactory/bin/sim build 2>&1 | tee make.log > >>> > >>> or > >>> > >>> export VERBOSE=yes > >>> make sim 2>&1 | tee make.log > >>> > >>> (depending on whether you used simfactory or not), please? > >>> > >>> Yours, > >>> Roland > >>> > >>>> Hello, > >>>> > >>>> Every time I try to compile my code I get the following error. > >>>> > >>>> COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c > >>>> /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic error: cannot open source file "mpi.h" > >>>> # include <mpi.h> > >>>> > >>>> This is new as of the most recent versions of Cactus. Previous compilations of the same code on the same cluster did not result in this error. Any help is appreciated. > >>>> > >>>> I used the config option: MPI_DIR = BUILD > >>>> > >>>> -DG > >>>> > >>>> Sent from my iPhone > >>>> _______________________________________________ > >>>> Users mailing list > >>>> Users(a)einsteintoolkit.org > >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users > >>> > >>> > >>> Yours, > >>> Roland > >>> > >>> > >>> -- > >>> My email is as private as my paper mail. I therefore support encrypting > >>> and signing email messages. Get my PGP key from http://pgp.mit.edu . > >> > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > _______________________________________________ > > Users mailing list > > Users(a)einsteintoolkit.org > > http://lists.einsteintoolkit.org/mailman/listinfo/users > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Going from lapacke.h to mkl_lapacke.h
by Chris Stevens 25 Mar '18

25 Mar '18

Hi everyone, I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library: -- In the local cluster, I had the option LAPACK_LIBS = lapack lapacke in my .ini file and used #include "lapacke.h" in the code. -- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64 LAPACK_LIBS = -mkl which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h". Question: Is this the right way to generalise to mkl lapacke? Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit. Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like LAPACKE_dgels -> MKL_LAPACKE_dgels Thanks! Chris -- Dr Chris Stevens Claude Leon Postdoctoral Fellow Department of Mathematics Rhodes University Room 5 Ph: +27 46 603 8932 Web: www.chrisdoesmaths.com <http://www.chrisdoesmaths.com>

2 4

Working group on Performance Optimization in the Einstein Toolkit
by Roland Haas 22 Mar '18

22 Mar '18

Hello all, during the ET meeting at NCSA in July / August 2017 we decided to form a working group on Performance Optimization in the Einstein Toolkit which Erik and I are tasked with leading. If you are currently working on performance optimization or have the firm intention of doing so in the near future then we would like to hear from you and recruit you for the working group. Right now we intend to have a short hangout call on Friday Mar 23rd 9:00am CDT to go over projects. A pool for times is here: https://www.when2meet.com/?6775184-Vk8PU in case many people would prefer a different time. An *early* draft of the working group document can be found on the ET wiki here: https://docs.einsteintoolkit.org/et-docs/Working_Group_on_Performance_Optim… which you are free to read and extend. The hangout will use this link: https://hangouts.google.com/call/8XRtmArNv5CRm_DJpZUNAAEE Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

Re: [Users] mpi.h
by Roland Haas 21 Mar '18

21 Mar '18

Hello DG, Joel, > /share/apps/openmpi-1.6.5/include/mpi.h is readable. I think the > problem is with Cactus not recognizing the options for some reason. That would have been my guess. Yet in configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it has: INC_DIRS += $(MPI_INC_DIRS) MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include and even if HDF5 itself did not bring in MPI (maybe it would not even though MPI shows up in its make.HDF5.deps file ...) IOHDF5Util (whose source file gave you the error) does explicitly depend on MPI through its configuration.ccl file which includes: OPTIONAL MPI which has me quite confused. Hence the request for the export VERBOSE=yes make fixedmhd output since that one will output the exact compiler line that Cactus executes and will tell us exactly if the proper -I option was included. Joel's solution (setting CC and CXX to mpicc and mpicxx) will also work on the cluster where you have a pre-compiled OpenMPI but won't work when you do actually compile OpenMPI inside of Cactus since mpicc naturally does not exist at the time it would be needed to compile OpenMPI itself. Yours, Roland > > > On Mar 21, 2018, at 9:52 AM, Roland Haas <rhaas(a)illinois.edu> wrote: > > > > Hello DG, > > > > hmm that all look fine to me. From the content of > > configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it > > is using a system installed MPI (openmpi-1.6.5 in /share/apps) > > which makes sense given that you pass > > MPI_DIR=/share/apps/openmpi-1.6.5 in your config-info (which is > > basically a copy of your option list). So MPI_DIR=BUILD was not > > used in the end (through an oversight or something else I guess). > > > > Since you are passing an explicit > > > > MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > > > > I would definitely myself try and see if indeed there is a file > > > > /share/apps/openmpi-1.6.5/include/mpi.h > > > > and that this file is readable to your user account (eg does > > less /share/apps/openmpi-1.6.5/include/mpi.h work?). > > > > This is just in case /share/apps/openmpi-1.6.5 was eg retired by the > > sysadmins. > > > > Yours, > > Roland > > > >> Hello, > >> > >> Please see below. I’m using UH’s Maxwell cluster and don’t use > >> simfactory. > >> > >> Thanks, > >> > >> -DG > >> > >>> On Mar 20, 2018, at 11:47 AM, Roland Haas > >>> <roland.haas(a)physics.gatech.edu> wrote: > >>> > >>> Hello DG, > >>> > >>> it could be that something failed while MPI was compiled. Would > >>> you mind attaching the following files from inside of your Cactus > >>> tree, please? > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.MPI.defn > >> > >> INC_DIRS += $(MPI_INC_DIRS) > >> INC_DIRS_F += $(MPI_LIB_DIRS) > >> MPI_BUILD = > >> MPI_INSTALL_DIR = > >> HWLOC_DIR = > >> CCTK_MPI = 1 > >> MPI_DIR = /share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include > >> MPI_LIB_DIRS = /share/apps/openmpi-1.6.5/lib > >> MPI_LIBS = mpi mpi_cxx open-rte open-pal dl nsl util > >> HAVE_CAPABILITY_MPI = 1 > >> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.defn > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.defn > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.defn > >> INC_DIRS += $(HDF5_INC_DIRS) > >> INC_DIRS_F += $(HDF5_INC_DIRS) > >> HDF5_BUILD = > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> LIBSZ_DIR = > >> LIBZ_DIR = > >> HDF5_INSTALL_DIR = > >> HDF5_DIR = /share/apps/hdf5-1.8.12 > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> HDF5_INC_DIRS > >> = /share/apps/hdf5-1.8.12/lib /share/apps/hdf5-1.8.12/include /share/apps/openmpi-1.6.5/include > >> HDF5_LIB_DIRS > >> = /share/apps/hdf5-1.8.12/lib /share/apps/openmpi-1.6.5/lib > >> HDF5_LIBS = hdf5hl_fortran hdf5_fortran hdf5_hl hdf5 z > >> mpi mpi_cxx open-rte open-pal dl nsl util m z > >> HAVE_CAPABILITY_HDF5 = 1 > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.deps > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.deps > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.deps > >> > >>> > >>> as well as > >>> > >>> confis/sim/config-info (as well as the option list that you > >>> used)? > >> > >> # CONFIGURATION : fixedmhd > >> # CONFIG-DATE : Mon Mar 19 01:07:33 2018 (GMT) > >> # CONFIG-HOST : cusco.hpcc.uh.edu > >> # CONFIG-STATUS : 0 > >> # CONFIG-OPTIONS : > >> CC=icc > >> CPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> CXX=icpc > >> CXX_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> C_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> F77=ifort > >> F77_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> F90=ifort > >> F90_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> FFLAGS=-O3 -fast -Mipa=fast,inline -tp=x64 > >> FFTW=yes > >> FFTW_DIR=/home/dgarriso/fftw3 > >> FFTW_LIBS=fftw3_threads fftw3_mpi $(MPI_LIBS) fftw3 m > >> FPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> HDF5_DIR=/share/apps/hdf5-1.8.12 > >> LAPACK=yes > >> LAPACK_DIR=/home/dgarriso/ifort64/lib > >> LAPACK_LIBS=acml ifcoremt_pic imf irc svml > >> MPI_DIR=/share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > >> MPI_LIBS=mpi mpi_cxx open-rte open-pal dl nsl util > >> MPI_LIB_DIRS=/share/apps/openmpi-1.6.5/lib > >> OPTIMISE=yes > >> PTHREADS=no > >> VERBOSE=yes > >> > >>> > >>> If possible also the full output of: > >>> > >>> export VERBOSE=yes > >>> simfactory/bin/sim build 2>&1 | tee make.log > >>> > >>> or > >>> > >>> export VERBOSE=yes > >>> make sim 2>&1 | tee make.log > >>> > >>> (depending on whether you used simfactory or not), please? > >>> > >>> Yours, > >>> Roland > >>> > >>>> Hello, > >>>> > >>>> Every time I try to compile my code I get the following error. > >>>> > >>>> COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c > >>>> /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic > >>>> error: cannot open source file "mpi.h" # include <mpi.h> > >>>> > >>>> This is new as of the most recent versions of Cactus. Previous > >>>> compilations of the same code on the same cluster did not result > >>>> in this error. Any help is appreciated. > >>>> > >>>> I used the config option: MPI_DIR = BUILD > >>>> > >>>> -DG > >>>> > >>>> Sent from my iPhone > >>>> _______________________________________________ > >>>> Users mailing list > >>>> Users(a)einsteintoolkit.org > >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users > >>> > >>> > >>> Yours, > >>> Roland > >>> > >>> > >>> -- > >>> My email is as private as my paper mail. I therefore support > >>> encrypting and signing email messages. Get my PGP key from > >>> http://pgp.mit.edu . > >> > > > > > > > > -- > > My email is as private as my paper mail. I therefore support > > encrypting and signing email messages. Get my PGP key from > > http://pgp.mit.edu . > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Re: [Users] mpi.h
by Roland Haas 21 Mar '18

21 Mar '18

Hello DG, hmm that all look fine to me. From the content of configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it is using a system installed MPI (openmpi-1.6.5 in /share/apps) which makes sense given that you pass MPI_DIR=/share/apps/openmpi-1.6.5 in your config-info (which is basically a copy of your option list). So MPI_DIR=BUILD was not used in the end (through an oversight or something else I guess). Since you are passing an explicit MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include I would definitely myself try and see if indeed there is a file /share/apps/openmpi-1.6.5/include/mpi.h and that this file is readable to your user account (eg does less /share/apps/openmpi-1.6.5/include/mpi.h work?). This is just in case /share/apps/openmpi-1.6.5 was eg retired by the sysadmins. Yours, Roland > Hello, > > Please see below. I’m using UH’s Maxwell cluster and don’t use simfactory. > > Thanks, > > -DG > > > On Mar 20, 2018, at 11:47 AM, Roland Haas <roland.haas(a)physics.gatech.edu> wrote: > > > > Hello DG, > > > > it could be that something failed while MPI was compiled. Would you mind attaching the following files from inside of your Cactus tree, please? > > > > configs/sim/bindings/Configuration/Capabilities/make.MPI.defn > > INC_DIRS += $(MPI_INC_DIRS) > INC_DIRS_F += $(MPI_LIB_DIRS) > MPI_BUILD = > MPI_INSTALL_DIR = > HWLOC_DIR = > CCTK_MPI = 1 > MPI_DIR = /share/apps/openmpi-1.6.5 > MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include > MPI_LIB_DIRS = /share/apps/openmpi-1.6.5/lib > MPI_LIBS = mpi mpi_cxx open-rte open-pal dl nsl util > HAVE_CAPABILITY_MPI = 1 > > > configs/sim/bindings/Configuration/Capabilities/make.HDF5.defn > > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.defn > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.defn > INC_DIRS += $(HDF5_INC_DIRS) > INC_DIRS_F += $(HDF5_INC_DIRS) > HDF5_BUILD = > HDF5_ENABLE_CXX = no > HDF5_ENABLE_FORTRAN = yes > LIBSZ_DIR = > LIBZ_DIR = > HDF5_INSTALL_DIR = > HDF5_DIR = /share/apps/hdf5-1.8.12 > HDF5_ENABLE_CXX = no > HDF5_ENABLE_FORTRAN = yes > HDF5_INC_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/hdf5-1.8.12/include /share/apps/openmpi-1.6.5/include > HDF5_LIB_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/openmpi-1.6.5/lib > HDF5_LIBS = hdf5hl_fortran hdf5_fortran hdf5_hl hdf5 z mpi mpi_cxx open-rte open-pal dl nsl util m z > HAVE_CAPABILITY_HDF5 = 1 > > > > configs/sim/bindings/Configuration/Capabilities/make.HDF5.deps > > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.deps > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.deps > > > > > as well as > > > > confis/sim/config-info (as well as the option list that you used)? > > # CONFIGURATION : fixedmhd > # CONFIG-DATE : Mon Mar 19 01:07:33 2018 (GMT) > # CONFIG-HOST : cusco.hpcc.uh.edu > # CONFIG-STATUS : 0 > # CONFIG-OPTIONS : > CC=icc > CPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > CXX=icpc > CXX_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > C_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > F77=ifort > F77_OPTIMISE_FLAGS=-Os -ip -no-prec-div > F90=ifort > F90_OPTIMISE_FLAGS=-Os -ip -no-prec-div > FFLAGS=-O3 -fast -Mipa=fast,inline -tp=x64 > FFTW=yes > FFTW_DIR=/home/dgarriso/fftw3 > FFTW_LIBS=fftw3_threads fftw3_mpi $(MPI_LIBS) fftw3 m > FPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > HDF5_DIR=/share/apps/hdf5-1.8.12 > LAPACK=yes > LAPACK_DIR=/home/dgarriso/ifort64/lib > LAPACK_LIBS=acml ifcoremt_pic imf irc svml > MPI_DIR=/share/apps/openmpi-1.6.5 > MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > MPI_LIBS=mpi mpi_cxx open-rte open-pal dl nsl util > MPI_LIB_DIRS=/share/apps/openmpi-1.6.5/lib > OPTIMISE=yes > PTHREADS=no > VERBOSE=yes > > > > > If possible also the full output of: > > > > export VERBOSE=yes > > simfactory/bin/sim build 2>&1 | tee make.log > > > > or > > > > export VERBOSE=yes > > make sim 2>&1 | tee make.log > > > > (depending on whether you used simfactory or not), please? > > > > Yours, > > Roland > > > >> Hello, > >> > >> Every time I try to compile my code I get the following error. > >> > >> COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c > >> /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic error: cannot open source file "mpi.h" > >> # include <mpi.h> > >> > >> This is new as of the most recent versions of Cactus. Previous compilations of the same code on the same cluster did not result in this error. Any help is appreciated. > >> > >> I used the config option: MPI_DIR = BUILD > >> > >> -DG > >> > >> Sent from my iPhone > >> _______________________________________________ > >> Users mailing list > >> Users(a)einsteintoolkit.org > >> http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > > > Yours, > > Roland > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users