Users March 2018

users@lists.einsteintoolkit.org

17 participants
26 discussions

Going from lapacke.h to mkl_lapacke.h
by Chris Stevens 25 Mar '18

25 Mar '18

Hi everyone, I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library: -- In the local cluster, I had the option LAPACK_LIBS = lapack lapacke in my .ini file and used #include "lapacke.h" in the code. -- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64 LAPACK_LIBS = -mkl which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h". Question: Is this the right way to generalise to mkl lapacke? Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit. Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like LAPACKE_dgels -> MKL_LAPACKE_dgels Thanks! Chris -- Dr Chris Stevens Claude Leon Postdoctoral Fellow Department of Mathematics Rhodes University Room 5 Ph: +27 46 603 8932 Web: www.chrisdoesmaths.com <http://www.chrisdoesmaths.com>

2 4

Working group on Performance Optimization in the Einstein Toolkit
by Roland Haas 22 Mar '18

22 Mar '18

Hello all, during the ET meeting at NCSA in July / August 2017 we decided to form a working group on Performance Optimization in the Einstein Toolkit which Erik and I are tasked with leading. If you are currently working on performance optimization or have the firm intention of doing so in the near future then we would like to hear from you and recruit you for the working group. Right now we intend to have a short hangout call on Friday Mar 23rd 9:00am CDT to go over projects. A pool for times is here: https://www.when2meet.com/?6775184-Vk8PU in case many people would prefer a different time. An *early* draft of the working group document can be found on the ET wiki here: https://docs.einsteintoolkit.org/et-docs/Working_Group_on_Performance_Optim… which you are free to read and extend. The hangout will use this link: https://hangouts.google.com/call/8XRtmArNv5CRm_DJpZUNAAEE Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

Re: [Users] mpi.h
by Roland Haas 21 Mar '18

21 Mar '18

Hello DG, Joel, > /share/apps/openmpi-1.6.5/include/mpi.h is readable. I think the > problem is with Cactus not recognizing the options for some reason. That would have been my guess. Yet in configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it has: INC_DIRS += $(MPI_INC_DIRS) MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include and even if HDF5 itself did not bring in MPI (maybe it would not even though MPI shows up in its make.HDF5.deps file ...) IOHDF5Util (whose source file gave you the error) does explicitly depend on MPI through its configuration.ccl file which includes: OPTIONAL MPI which has me quite confused. Hence the request for the export VERBOSE=yes make fixedmhd output since that one will output the exact compiler line that Cactus executes and will tell us exactly if the proper -I option was included. Joel's solution (setting CC and CXX to mpicc and mpicxx) will also work on the cluster where you have a pre-compiled OpenMPI but won't work when you do actually compile OpenMPI inside of Cactus since mpicc naturally does not exist at the time it would be needed to compile OpenMPI itself. Yours, Roland > > > On Mar 21, 2018, at 9:52 AM, Roland Haas <rhaas(a)illinois.edu> wrote: > > > > Hello DG, > > > > hmm that all look fine to me. From the content of > > configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it > > is using a system installed MPI (openmpi-1.6.5 in /share/apps) > > which makes sense given that you pass > > MPI_DIR=/share/apps/openmpi-1.6.5 in your config-info (which is > > basically a copy of your option list). So MPI_DIR=BUILD was not > > used in the end (through an oversight or something else I guess). > > > > Since you are passing an explicit > > > > MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > > > > I would definitely myself try and see if indeed there is a file > > > > /share/apps/openmpi-1.6.5/include/mpi.h > > > > and that this file is readable to your user account (eg does > > less /share/apps/openmpi-1.6.5/include/mpi.h work?). > > > > This is just in case /share/apps/openmpi-1.6.5 was eg retired by the > > sysadmins. > > > > Yours, > > Roland > > > >> Hello, > >> > >> Please see below. I’m using UH’s Maxwell cluster and don’t use > >> simfactory. > >> > >> Thanks, > >> > >> -DG > >> > >>> On Mar 20, 2018, at 11:47 AM, Roland Haas > >>> <roland.haas(a)physics.gatech.edu> wrote: > >>> > >>> Hello DG, > >>> > >>> it could be that something failed while MPI was compiled. Would > >>> you mind attaching the following files from inside of your Cactus > >>> tree, please? > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.MPI.defn > >> > >> INC_DIRS += $(MPI_INC_DIRS) > >> INC_DIRS_F += $(MPI_LIB_DIRS) > >> MPI_BUILD = > >> MPI_INSTALL_DIR = > >> HWLOC_DIR = > >> CCTK_MPI = 1 > >> MPI_DIR = /share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include > >> MPI_LIB_DIRS = /share/apps/openmpi-1.6.5/lib > >> MPI_LIBS = mpi mpi_cxx open-rte open-pal dl nsl util > >> HAVE_CAPABILITY_MPI = 1 > >> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.defn > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.defn > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.defn > >> INC_DIRS += $(HDF5_INC_DIRS) > >> INC_DIRS_F += $(HDF5_INC_DIRS) > >> HDF5_BUILD = > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> LIBSZ_DIR = > >> LIBZ_DIR = > >> HDF5_INSTALL_DIR = > >> HDF5_DIR = /share/apps/hdf5-1.8.12 > >> HDF5_ENABLE_CXX = no > >> HDF5_ENABLE_FORTRAN = yes > >> HDF5_INC_DIRS > >> = /share/apps/hdf5-1.8.12/lib /share/apps/hdf5-1.8.12/include /share/apps/openmpi-1.6.5/include > >> HDF5_LIB_DIRS > >> = /share/apps/hdf5-1.8.12/lib /share/apps/openmpi-1.6.5/lib > >> HDF5_LIBS = hdf5hl_fortran hdf5_fortran hdf5_hl hdf5 z > >> mpi mpi_cxx open-rte open-pal dl nsl util m z > >> HAVE_CAPABILITY_HDF5 = 1 > >>> > >>> configs/sim/bindings/Configuration/Capabilities/make.HDF5.deps > >> > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.deps > >> include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.deps > >> > >>> > >>> as well as > >>> > >>> confis/sim/config-info (as well as the option list that you > >>> used)? > >> > >> # CONFIGURATION : fixedmhd > >> # CONFIG-DATE : Mon Mar 19 01:07:33 2018 (GMT) > >> # CONFIG-HOST : cusco.hpcc.uh.edu > >> # CONFIG-STATUS : 0 > >> # CONFIG-OPTIONS : > >> CC=icc > >> CPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> CXX=icpc > >> CXX_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> C_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > >> F77=ifort > >> F77_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> F90=ifort > >> F90_OPTIMISE_FLAGS=-Os -ip -no-prec-div > >> FFLAGS=-O3 -fast -Mipa=fast,inline -tp=x64 > >> FFTW=yes > >> FFTW_DIR=/home/dgarriso/fftw3 > >> FFTW_LIBS=fftw3_threads fftw3_mpi $(MPI_LIBS) fftw3 m > >> FPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > >> HDF5_DIR=/share/apps/hdf5-1.8.12 > >> LAPACK=yes > >> LAPACK_DIR=/home/dgarriso/ifort64/lib > >> LAPACK_LIBS=acml ifcoremt_pic imf irc svml > >> MPI_DIR=/share/apps/openmpi-1.6.5 > >> MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > >> MPI_LIBS=mpi mpi_cxx open-rte open-pal dl nsl util > >> MPI_LIB_DIRS=/share/apps/openmpi-1.6.5/lib > >> OPTIMISE=yes > >> PTHREADS=no > >> VERBOSE=yes > >> > >>> > >>> If possible also the full output of: > >>> > >>> export VERBOSE=yes > >>> simfactory/bin/sim build 2>&1 | tee make.log > >>> > >>> or > >>> > >>> export VERBOSE=yes > >>> make sim 2>&1 | tee make.log > >>> > >>> (depending on whether you used simfactory or not), please? > >>> > >>> Yours, > >>> Roland > >>> > >>>> Hello, > >>>> > >>>> Every time I try to compile my code I get the following error. > >>>> > >>>> COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c > >>>> /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic > >>>> error: cannot open source file "mpi.h" # include <mpi.h> > >>>> > >>>> This is new as of the most recent versions of Cactus. Previous > >>>> compilations of the same code on the same cluster did not result > >>>> in this error. Any help is appreciated. > >>>> > >>>> I used the config option: MPI_DIR = BUILD > >>>> > >>>> -DG > >>>> > >>>> Sent from my iPhone > >>>> _______________________________________________ > >>>> Users mailing list > >>>> Users(a)einsteintoolkit.org > >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users > >>> > >>> > >>> Yours, > >>> Roland > >>> > >>> > >>> -- > >>> My email is as private as my paper mail. I therefore support > >>> encrypting and signing email messages. Get my PGP key from > >>> http://pgp.mit.edu . > >> > > > > > > > > -- > > My email is as private as my paper mail. I therefore support > > encrypting and signing email messages. Get my PGP key from > > http://pgp.mit.edu . > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Re: [Users] mpi.h
by Roland Haas 21 Mar '18

21 Mar '18

Hello DG, hmm that all look fine to me. From the content of configs/sim/bindings/Configuration/Capabilities/make.MPI.defn it is using a system installed MPI (openmpi-1.6.5 in /share/apps) which makes sense given that you pass MPI_DIR=/share/apps/openmpi-1.6.5 in your config-info (which is basically a copy of your option list). So MPI_DIR=BUILD was not used in the end (through an oversight or something else I guess). Since you are passing an explicit MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include I would definitely myself try and see if indeed there is a file /share/apps/openmpi-1.6.5/include/mpi.h and that this file is readable to your user account (eg does less /share/apps/openmpi-1.6.5/include/mpi.h work?). This is just in case /share/apps/openmpi-1.6.5 was eg retired by the sysadmins. Yours, Roland > Hello, > > Please see below. I’m using UH’s Maxwell cluster and don’t use simfactory. > > Thanks, > > -DG > > > On Mar 20, 2018, at 11:47 AM, Roland Haas <roland.haas(a)physics.gatech.edu> wrote: > > > > Hello DG, > > > > it could be that something failed while MPI was compiled. Would you mind attaching the following files from inside of your Cactus tree, please? > > > > configs/sim/bindings/Configuration/Capabilities/make.MPI.defn > > INC_DIRS += $(MPI_INC_DIRS) > INC_DIRS_F += $(MPI_LIB_DIRS) > MPI_BUILD = > MPI_INSTALL_DIR = > HWLOC_DIR = > CCTK_MPI = 1 > MPI_DIR = /share/apps/openmpi-1.6.5 > MPI_INC_DIRS = /share/apps/openmpi-1.6.5/include > MPI_LIB_DIRS = /share/apps/openmpi-1.6.5/lib > MPI_LIBS = mpi mpi_cxx open-rte open-pal dl nsl util > HAVE_CAPABILITY_MPI = 1 > > > configs/sim/bindings/Configuration/Capabilities/make.HDF5.defn > > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.defn > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.defn > INC_DIRS += $(HDF5_INC_DIRS) > INC_DIRS_F += $(HDF5_INC_DIRS) > HDF5_BUILD = > HDF5_ENABLE_CXX = no > HDF5_ENABLE_FORTRAN = yes > LIBSZ_DIR = > LIBZ_DIR = > HDF5_INSTALL_DIR = > HDF5_DIR = /share/apps/hdf5-1.8.12 > HDF5_ENABLE_CXX = no > HDF5_ENABLE_FORTRAN = yes > HDF5_INC_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/hdf5-1.8.12/include /share/apps/openmpi-1.6.5/include > HDF5_LIB_DIRS = /share/apps/hdf5-1.8.12/lib /share/apps/openmpi-1.6.5/lib > HDF5_LIBS = hdf5hl_fortran hdf5_fortran hdf5_hl hdf5 z mpi mpi_cxx open-rte open-pal dl nsl util m z > HAVE_CAPABILITY_HDF5 = 1 > > > > configs/sim/bindings/Configuration/Capabilities/make.HDF5.deps > > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.MPI.deps > include /home/dgarriso/Cactus/configs/fixedmhd/bindings/Configuration/Capabilities/make.ZLIB.deps > > > > > as well as > > > > confis/sim/config-info (as well as the option list that you used)? > > # CONFIGURATION : fixedmhd > # CONFIG-DATE : Mon Mar 19 01:07:33 2018 (GMT) > # CONFIG-HOST : cusco.hpcc.uh.edu > # CONFIG-STATUS : 0 > # CONFIG-OPTIONS : > CC=icc > CPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > CXX=icpc > CXX_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > C_OPTIMISE_FLAGS=-Os -ip -pthread -no-prec-div > F77=ifort > F77_OPTIMISE_FLAGS=-Os -ip -no-prec-div > F90=ifort > F90_OPTIMISE_FLAGS=-Os -ip -no-prec-div > FFLAGS=-O3 -fast -Mipa=fast,inline -tp=x64 > FFTW=yes > FFTW_DIR=/home/dgarriso/fftw3 > FFTW_LIBS=fftw3_threads fftw3_mpi $(MPI_LIBS) fftw3 m > FPPFLAGS=-DCCTK_DISABLE_OMP_COLLAPSE > HDF5_DIR=/share/apps/hdf5-1.8.12 > LAPACK=yes > LAPACK_DIR=/home/dgarriso/ifort64/lib > LAPACK_LIBS=acml ifcoremt_pic imf irc svml > MPI_DIR=/share/apps/openmpi-1.6.5 > MPI_INC_DIRS=/share/apps/openmpi-1.6.5/include > MPI_LIBS=mpi mpi_cxx open-rte open-pal dl nsl util > MPI_LIB_DIRS=/share/apps/openmpi-1.6.5/lib > OPTIMISE=yes > PTHREADS=no > VERBOSE=yes > > > > > If possible also the full output of: > > > > export VERBOSE=yes > > simfactory/bin/sim build 2>&1 | tee make.log > > > > or > > > > export VERBOSE=yes > > make sim 2>&1 | tee make.log > > > > (depending on whether you used simfactory or not), please? > > > > Yours, > > Roland > > > >> Hello, > >> > >> Every time I try to compile my code I get the following error. > >> > >> COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c > >> /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic error: cannot open source file "mpi.h" > >> # include <mpi.h> > >> > >> This is new as of the most recent versions of Cactus. Previous compilations of the same code on the same cluster did not result in this error. Any help is appreciated. > >> > >> I used the config option: MPI_DIR = BUILD > >> > >> -DG > >> > >> Sent from my iPhone > >> _______________________________________________ > >> Users mailing list > >> Users(a)einsteintoolkit.org > >> http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > > > Yours, > > Roland > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

mpi.h
by David Garrison 21 Mar '18

21 Mar '18

Hello, Every time I try to compile my code I get the following error. COMPILING arrangements/CactusPUGHIO/IOHDF5Util/src/Startup.c /share/apps/hdf5-1.8.12/include/H5public.h(61): catastrophic error: cannot open source file "mpi.h" # include <mpi.h> This is new as of the most recent versions of Cactus. Previous compilations of the same code on the same cluster did not result in this error. Any help is appreciated. I used the config option: MPI_DIR = BUILD -DG Sent from my iPhone

3 3

Grid Consistency Failure
by Tamanna Jain 20 Mar '18

20 Mar '18

Hey all, I am new to Einstein Toolkit,I was running a parameter file for BBH but it crashed after elapsing 23M time with the error "grid consistency check failed". i have attached error and parameter file along with it.Help me,what should i do. -- Regards Tamanna Jain

2 1

Trouble compiling ET with MacPorts: stdlib.h
by Bernard Kelly 19 Mar '18

19 Mar '18

Hi. I'm having trouble compiling the ET (current release and previous) on Mac OS X with MacPorts. I'm using the configuration options in simfactory/mdb/optionlists/osx-macports.cfg Initial make sim-config goes fine, but when I start the actual make, I hit a problem in AHFinderDirect: ------------------------------ COMPILING arrangements/EinsteinAnalysis/AHFinderDirect/src/jtutil/array.cc In file included from /opt/local/include/gcc/c++/bits/std_abs.h:38:0, from /opt/local/include/gcc/c++/cmath:47, from /opt/local/include/gcc/c++/complex:44, from /Users/bjkelly1/nobackup/codes/Cactus/src/include/cctk_Types.h:50, from /Users/bjkelly1/nobackup/codes/Cactus/configs/sim/bindings/include/AHFinderDirect/cctk_Arguments.h:14, from /Users/bjkelly1/nobackup/codes/Cactus/configs/sim/bindings/include/AHFinderDirect/cctk_ScheduleFunctions.h:1, from /Users/bjkelly1/nobackup/codes/Cactus/configs/sim/bindings/include/AHFinderDirect/cctk.h:6, from /Users/bjkelly1/nobackup/codes/Cactus/arrangements/EinsteinAnalysis/AHFinderDirect/src/jtutil/array.cc:29: /opt/local/include/gcc7/c++/stdlib.h:30:16: fatal error: stdlib.h: No such file or directory # include_next <stdlib.h> ------------------------------ The file /opt/local/include/gcc7/c++/stdlib.h definitely exists; I think it's the "include_next" directive that's causing the problem. Is it looking for a similarly named header file somewhere else? Ideas/solutions welcome. Thanks! Bernard

2 2

cannot access trac.einsteintoolkit.org using Firefox
by Roland Haas 19 Mar '18

19 Mar '18

Hello all, I seem to be unable to access trac.einsteintoolkit.org using Firefox (Debian ESR 52.5.0) which when trying to access https://trac.einsteintoolkit.org/ shows --8<-- Unable to connect Firefox can’t establish a connection to the server at trac.einsteintoolkit.org. The site could be temporarily unavailable or too busy. Try again in a few moments. If you are unable to load any pages, check your computer’s network connection. If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web. --8<-- which it goes to even when entering http://trac.einsteintoolkit.org into the address bar. With Chromium things seem to work if one points it at http://trac.einsteintoolkit.org but not at https://trac.einsteintoolkit.org . Since I will have to log in I would really very much like the ssl encrypted page to work. If this actually is a server issue, could someone at LSU please look into this, please? Frank's monitor script shows an incorrect certificate for the server: https://www.cct.lsu.edu/~knarf/cgi-bin/monitor.cgi Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 3

Grid Consistency Failure
by Tamanna Jain 19 Mar '18

19 Mar '18

2 5

meeting minutes for 2018-03-19
by Roland Haas 19 Mar '18

19 Mar '18

Present: Roland, Bill, Peter, Sam, Zach, Qian, Steve, Gabrielle, Peter, Antoni Ramos Buades, Bhavesh, Yosef Trac update: * has been updated and is working but cannot be reached due to port 443 * should be resolved soon this morning * as of now server is up but needs a security exception to view it * status icons for Jenkins are not yet working (a cross site scripting issue) Working groups: * want first round of status updates * check on SI2 related working groups * currently have 3 established groups, have 10 planned groups * Roland to (finally) create performance upgrades one * Steve to create maintainers working group * Initial data: ping Josh, Roland to ping people in general * aim for reports in 2 weeks, aim to have reports on the first Monday of each month Code reviews for thorns to be included: * tickets https://trac.einsteintoolkit.org/ticket/2101 https://trac.einsteintoolkit.org/ticket/1974 https://trac.einsteintoolkit.org/ticket/2064 https://trac.einsteintoolkit.org/ticket/2042 * include in master thorn list once they have a test and compile, remove if they break, remove from release if not reviewed US ET workshop: * had first SOC call on Friday * discussed schedule layout and facilities, draft of website * discussed split between invited and contributed talks Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users March 2018