Users March 2018

users@lists.einsteintoolkit.org

17 participants
26 discussions

Einstein Toolkit Meeting Reminder
by Steven R. Brandt 31 Oct '19

31 Oct '19

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --Steve

8 94

Re: [Users] [ET Trac] [Einstein Toolkit] #1326: running loopcontrol on strange number of threads fails
by Erik Schnetter 11 Apr '19

11 Apr '19

Do you have a backtrace? -erik On Sunday, March 22, 2015, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #1326: running loopcontrol on strange number of threads fails > ------------------------------------+--------------------------------------- > Reporter: rhaas | Owner: > Type: defect | Status: new > Priority: minor | Milestone: > Component: EinsteinToolkit thorn | Version: > Resolution: | Keywords: LoopControl > ------------------------------------+--------------------------------------- > > Comment (by rhaas): > > This still happens even with current (Sun Mar 22 18:46:21 CET 2015) trunk, > though failure looks a bit different now: > {{{ > actus_sim: > /data/rhaas/postdoc/gr/ET_trunk/configs/sim/build/LoopControl/loopcontrol.cc:312: > T {anonymous}::divexact(T, T) [with T = int]: Assertion `i % j == 0' > failed > }}} > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/1326#comment:1> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Agenda for Monday's Meeting
by Steven R. Brandt 30 May '18

30 May '18

Please update the Wiki with agenda items for Monday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --Steve

2 32

Question on profile code
by Hyun Lim 27 Apr '18

27 Apr '18

Dear all, Does anyone have any experiences about profiling ET with vampirtrace? Any recommendations how to profile ET? Hyun

2 1

Carpet documentation?
by Jonah Miller 30 Mar '18

30 Mar '18

Hello all, I seem to recall that there was documentation for setting up refinement in Carpet---i.e., what parameters like Shiftout and centering do the grid setup. I thought it was one of the ThornDoc files for Carpet, but I can no longer find it. Does anyone know where it might be? Thanks! Best, Jonah Miller

2 2

Generalising to OpenMP
by Chris Stevens 29 Mar '18

29 Mar '18

Hi everyone, I am new to OpenMP programming, and have quickly come across an (I think) interesting task that I want to parallelise with OpenMP that does not seem straightforwardly generalisable. I would like to hear your feedback on the potential solutions and whether there is a better way to do it: Task: Create a scratch array, say of CCTK_REAL, during an initialisation phase that is shared with other code using a header file. This array is defined only once. This scratch array is then used as temporary storage space in a function that does some computation within a for loop. i.e. function blah () { for (....) { computation involving scratch array } } This works fine when OpenMP is turned off. However when you add to the for loop above #pragma omp parallel for, then one sees that this one scratch array will be being used by multiple threads at once, thus causing issues. Solution 1: Each thread has its own scratch space. I am not sure if it is within the OpenMP paradigm to create scratch spaces for each thread within the initialisation file, and share them through a header. This doesn't seem to work and I am thinking this shouldn't be possible? Solution 2: Create one serial scratch space during initialisation that has size dependent on omp_get_num_threads(). One would then have to change the computation involving the scratch array to be dependent on omp_get_thread_num() so as to use its allocated part of the scratch space. Solution 3: One could do something like #pragma omp parallel { create scratch space #pragma omp parallel for for { computation involving thread specific scratch space } } In summary: Solution 1 doesn't seem to work and probably doesn't fit into the OpenMP paradigm. Solution 2 would work but doesn't seem very nice. Solution 3 is nicer but inefficient as I don't want to be creating scratch spaces all the time. Is there perhaps another way that fits better with the OpenMP paradigm? Many thanks! Chris -- Dr Chris Stevens Claude Leon Postdoctoral Fellow Department of Mathematics Rhodes University Room 5 Ph: +27 46 603 8932 Web: www.chrisdoesmaths.com <http://www.chrisdoesmaths.com>

2 3

Error in output of helloworld
by Swarnim Shashank 27 Mar '18

27 Mar '18

Hello, I am an undergraduate student and I am learning ET for my project. I am having trouble running the helloworld simulation. Everything works out but when I do "show-output" I get this on my terminal: Simulation name: helloworld Warning: job status is U Warning: job status is U Warning: Job is not running, cannot retreive exechost ================================================================================ The job's Formaline output is: ================================================================================ (file does not exist) ================================================================================ The job's stdout is: ================================================================================ Simulation name: helloworld Running simulation helloworld Preparing: Checking: /home/swarnim/simulations/helloworld/output-0000-active koszmikroid-X550LD Mon Mar 26 17:44:46 IST 2018 Environment: Starting: INFO (Cactus): Increasing logging level from 0 to 3 Mon Mar 26 17:44:47 IST 2018 Simfactory Done at date: 0 ================================================================================ The job's stderr is: ================================================================================ + set -e + cd /home/swarnim/simulations/helloworld/output-0000-active + echo Checking: + pwd + hostname + date + echo Environment: + export CACTUS_NUM_PROCS=1 + export CACTUS_NUM_THREADS=2 + export GMON_OUT_PREFIX=gmon.out + export OMP_NUM_THREADS=2 + env + sort + echo Starting: + date +%s + export CACTUS_STARTTIME=1522066486 + [ 1 = 1 ] + [ 0 -eq 0 ] + /home/swarnim/simulations/helloworld/SIMFACTORY/exe/cactus_sim -L 3 /home/swarnim/simulations/helloworld/output-0000/HelloWorld.par Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: PMPI_Comm_rank(110): MPI_Comm_rank(comm=0xe0cc28e0, rank=0x7ffd2aa27d38) failed PMPI_Comm_rank(68).: Invalid communicator [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=873024773 : system msg for write_line failure : Bad file descriptor ================================================================================ Please let me know what the problem is and how to sort it. Thank you Sincerely *Swarnim Shashank* Fourth Year Integrated MSc. Student UM-DAE-Centre for Excellence in Basic Sciences Mumbai

2 1

meeting minutes for 2018-03-26
by Roland Haas 26 Mar '18

26 Mar '18

Present: Roland, Helvi, Peter, Sam, Steve, Bhavesh, Zach, Antoni, Qian Einstein Toolkit Maintainers: * list is outdated: http://einsteintoolkit.org/maintainers-credits.html ** add Steve Brandt ** update status of persons SSL status of ET websites: * apache config files seem to indicate that the correct certs should be in use but the monitor shows otherwise * Steve is working on this with LSU computer support Suggest to move to bitbucket: * would relieve burden from Steve to be sole website maintainer * would require ssl ready redirect from https;//www.einsteintoolkit.org since we cannot directly point the DNS entry to the bitbucket site (cert would not match) * User and Reference documentation do not work yet * please poke around in website to find outdated information Performance working group: * try to set up interaction with scheduling working group Suggestion: * evaluate need / work required for a full discontinuous Galerkin method driver with HP refinement ReCAPTCHA issue: * currently no captchas are required to edit the page * Steve is working on fixing it before the deadline Wiki page: * add direct link to landing page of the ET: approved US ET meeting: * in process of securing hosting at GT, Bhavesh will send emails once hosting is sorted out Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

performance optimization working group
by Roland Haas 25 Mar '18

25 Mar '18

Dear all, after the discussions in the Einstein Toolkit workshop at NCSA we have formed a new working group "Performance optimization" led by Erik and myself. This group will serve as an arena to coordinate activities to improve the Einstein Toolkit performance through code optimization, algorithmic improvements and code generation to enable new and exciting physical applications in the Einstein Toolkit. You can find a more detailed description of the working group here https://docs.einsteintoolkit.org/et-docs/Working_Group_on_Performance_Optim… If you are interested in joining the working group please sign up for our mailing list performanceoptimization-wg(a)einsteintoolkit.org here http://lists.einsteintoolkit.org/mailman/listinfo/performanceoptimization-wg The working group document will be presented to the ET maintainers in the ET call on Monday March 26 2018 and if all goes well will be approved a week after that. Best wishes, Roland, Erik, Zach -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Re: [Users] [External - Warning: Suspect Attachment] null Re: mpi.h 9%
by Roland Haas 25 Mar '18

25 Mar '18

Hello DG, it looks like you are trying to use the system mpicc to compile OpenMPI in Cactus' ExternalLibraries. This is a very odd thing to do and I am not surprised that you would fail attempting this. Instead why not set MPI_DIR = <PATH-TO-DIRECTORY-WHERE-MPICC-IC>/.. ie the parent directory of the system MPI's "bin" directory? Leave the other MPI settings blank and Cactus should obtain the required settings from the system MPI that it will find in MPI_DIR. Yours, Roland > Hello, > > Using mpicc, etc… seems to work a lot better but I’m still having a couple issues. When I use the cluster’s mpi, I get the error > > ld: cannot find -lmpi_usempif08 > make[2]: *** [/home/dgarriso/Cactus/exe/cactus_fixedmhd] Error 1 > make[1]: *** [fixedmhd] Error 2 > make: *** [fixedmhd-rebuild] Error 2 > > When I try to build the Cactus mpi, it fails > > Making all in libevent > Making all in . > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT event.lo -MD -MP -MF .deps/event.Tpo -c event.c -o event.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evthread.lo -MD -MP -MF .deps/evthread.Tpo -c evthread.c -o evthread.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evmap.lo -MD -MP -MF .deps/evmap.Tpo -c evmap.c -o evmap.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT log.lo -MD -MP -MF .deps/log.Tpo -c log.c -o log.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evutil.lo -MD -MP -MF .deps/evutil.Tpo -c evutil.c -o evutil.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT evutil_rand.lo -MD -MP -MF .deps/evutil_rand.Tpo -c evutil_rand.c -o evutil_rand.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT strlcpy.lo -MD -MP -MF .deps/strlcpy.Tpo -c strlcpy.c -o strlcpy.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT select.lo -MD -MP -MF .deps/select.Tpo -c select.c -o select.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT poll.lo -MD -MP -MF .deps/poll.Tpo -c poll.c -o poll.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT epoll.lo -MD -MP -MF .deps/epoll.Tpo -c epoll.c -o epoll.o > libtool: compile: mpiicc -DHAVE_CONFIG_H -I. -I./compat -I./include -I./include -DCCTK_DISABLE_OMP_COLLAPSE -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -std=c99 -Os -ip -pthread -no-prec-div -Wall -fno-strict-aliasing -MT signal.lo -MD -MP -MF .deps/signal.Tpo -c signal.c -o signal.o > signal.c(135): error: identifier "NSIG" is undefined > int ncaught[NSIG]; > ^ > > signal.c(253): error: identifier "SA_RESTART" is undefined > sa.sa_flags |= SA_RESTART; > ^ > > signal.c(281): error: identifier "NSIG" is undefined > EVUTIL_ASSERT(evsignal >= 0 && evsignal < NSIG); > ^ > > signal.c(355): error: identifier "NSIG" is undefined > EVUTIL_ASSERT(evsignal >= 0 && evsignal < NSIG); > ^ > > signal.c(409): error: identifier "NSIG" is undefined > for (i = 0; i < NSIG; ++i) { > ^ > > compilation aborted for signal.c (code 2) > make[10]: *** [signal.lo] Error 1 > make[9]: *** [all-recursive] Error 1 > make[8]: *** [all] Error 2 > make[7]: *** [all-recursive] Error 1 > make[6]: *** [all-recursive] Error 1 > make[5]: *** [all-recursive] Error 1 > Died at /home/dgarriso/Cactus/arrangements/ExternalLibraries/MPI/src/build.pl line 74. > make[4]: *** [/home/dgarriso/Cactus/configs/fixedmhd/scratch/done/MPI] Error 17 > make[3]: *** [make.checked] Error 2 > make[2]: *** [/home/dgarriso/Cactus/configs/fixedmhd/lib/libthorn_MPI.a] Error 2 > make[1]: *** [fixedmhd] Error 2 > make: *** [fixedmhd-rebuild] Error 2 > > Is this an issue with the cluster mpi installation or with how cactus works with the compilers? I am using Intel compilers from 2015 (intel/ics-2015-64-bit), and openmpi version 1.6.5 on the cluster compiled with intel compilers. > > -DG > > > On Mar 21, 2018, at 9:52 AM, Roland Haas <rhaas(a)illinois.edu> wrote: > > > > Hello DG, > > > > hmm that all look fine to me. From the content of > > configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it is using a system installed MPI (openmpi-1.6.5 in /share/apps) which makes sense given that you pass MPI_DIR=/share/apps/openmpi-1.6.5 in your config-info (which is basically a copy of your option list). So MPI_DIR=BUILD was not used in the end (through an oversight or something else I guess). > > > > Since you are passing an explicit > > > > MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > > > > I would definitely myself try and see if indeed there is a file > > > > /share/apps/openmpi-1.6.5/include/mpi.h > > > > and that this file is readable to your user account (eg does > > less /share/apps/openmpi-1.6.5/include/mpi.h work?). > > > > This is just in case /share/apps/openmpi-1.6.5 was eg retired by the > > sysadmins. > > > > Yours, > > Roland > > > >> Hello, > >> > >> Please see below. I’m using UH’s Maxwell cluster and don’t use simfactory. > >> > >> Thanks, > >> > >> -DG > >> > >>> On Mar 20, 2018, at 11:47 AM, Roland Haas <roland.haas(a)physics.gatech.edu> wrote: > >>> > >>> Hello DG, > >>> > >>> it could be that something failed while MPI was compiled. Would you mind attaching the following files from inside of your Cactus tree, please? > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.MPI.defn > >> > >> INC_DIRS += $(MPI_INC_DIRS) > >> INC_DIRS_F += $(MPI_LIB_DIRS) > >> MPI_BUILD = > >> MPI_INSTALL_DIR = > >> HWLOC_DIR = > >> CCTK_MPI = 1 > >> MPI_DIR = /share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include > >> MPI_LIB_DIRS = /share/apps/openmpi-1.6.5/lib > >> MPI_LIBS = mpi mpi_cxx open-rte open-pal dl nsl util > >> HAVE_CAPABILITY_MPI = 1 > >> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.defn > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.defn > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.defn > >> INC_DIRS += $(HDF5_INC_DIRS) > >> INC_DIRS_F += $(HDF5_INC_DIRS) > >> HDF5_BUILD = > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> LIBSZ_DIR = > >> LIBZ_DIR = > >> HDF5_INSTALL_DIR = > >> HDF5_DIR = /share/apps/hdf5-1.8.12 > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> HDF5_INC_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/hdf5-1.8.12/include /share/apps/openmpi-1.6.5/include > >> HDF5_LIB_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/openmpi-1.6.5/lib > >> HDF5_LIBS = hdf5hl_fortran hdf5_fortran hdf5_hl hdf5 z mpi mpi_cxx open-rte open-pal dl nsl util m z > >> HAVE_CAPABILITY_HDF5 = 1 > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.deps > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.deps > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.deps > >> > >>> > >>> as well as > >>> > >>> confis/sim/config-info (as well as the option list that you used)? > >> > >> # CONFIGURATION : fixedmhd > >> # CONFIG-DATE : Mon Mar 19 01:07:33 2018 (GMT) > >> # CONFIG-HOST : cusco.hpcc.uh.edu > >> # CONFIG-STATUS : 0 > >> # CONFIG-OPTIONS : > >> CC=icc > >> CPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> CXX=icpc > >> CXX_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> C_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> F77=ifort > >> F77_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> F90=ifort > >> F90_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> FFLAGS=-O3 -fast -Mipa=fast,inline -tp=x64 > >> FFTW=yes > >> FFTW_DIR=/home/dgarriso/fftw3 > >> FFTW_LIBS=fftw3_threads fftw3_mpi $(MPI_LIBS) fftw3 m > >> FPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> HDF5_DIR=/share/apps/hdf5-1.8.12 > >> LAPACK=yes > >> LAPACK_DIR=/home/dgarriso/ifort64/lib > >> LAPACK_LIBS=acml ifcoremt_pic imf irc svml > >> MPI_DIR=/share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > >> MPI_LIBS=mpi mpi_cxx open-rte open-pal dl nsl util > >> MPI_LIB_DIRS=/share/apps/openmpi-1.6.5/lib > >> OPTIMISE=yes > >> PTHREADS=no > >> VERBOSE=yes > >> > >>> > >>> If possible also the full output of: > >>> > >>> export VERBOSE=yes > >>> simfactory/bin/sim build 2>&1 | tee make.log > >>> > >>> or > >>> > >>> export VERBOSE=yes > >>> make sim 2>&1 | tee make.log > >>> > >>> (depending on whether you used simfactory or not), please? > >>> > >>> Yours, > >>> Roland > >>> > >>>> Hello, > >>>> > >>>> Every time I try to compile my code I get the following error. > >>>> > >>>> COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c > >>>> /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic error: cannot open source file "mpi.h" > >>>> # include <mpi.h> > >>>> > >>>> This is new as of the most recent versions of Cactus. Previous compilations of the same code on the same cluster did not result in this error. Any help is appreciated. > >>>> > >>>> I used the config option: MPI_DIR = BUILD > >>>> > >>>> -DG > >>>> > >>>> Sent from my iPhone > >>>> _______________________________________________ > >>>> Users mailing list > >>>> Users(a)einsteintoolkit.org > >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users > >>> > >>> > >>> Yours, > >>> Roland > >>> > >>> > >>> -- > >>> My email is as private as my paper mail. I therefore support encrypting > >>> and signing email messages. Get my PGP key from http://pgp.mit.edu . > >> > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > _______________________________________________ > > Users mailing list > > Users(a)einsteintoolkit.org > > http://lists.einsteintoolkit.org/mailman/listinfo/users > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users March 2018