[Users] config file for lonestar

[Yosef Zlochower]

Ian Hinder wrote:
> On 18 Mar 2011, at 17:30, Erik Schnetter wrote:
> 
>> That probably means it's not a startup problem... but it could still
>> be something strange with MPI, e.g. during regridding.
>>
>> Another issue I just remember is running out of memory.
> 
> Hi Yosef,
> 
> Also, it often helps to use OpenMP, since this reduces the number of MPI processes which need to communicate initially.  In your config it is enabled, but are you using it?
> 

Hi,

  Yes, I was using OpenMP

>> -erik
>>
>> On Fri, Mar 18, 2011 at 12:23 PM, Yosef Zlochower <yosef at astro.rit.edu> wrote:
>>> Thanks,
>>>
>>> Erik Schnetter wrote:
>>>
>>>> Yosef
>>>>
>>>> Yes, there is a "standard" config file, but I have not tested it on
>>>> large numbers of cores. I attach it below.
>>>>
>>>> If there is a problem on large numbers of cores, then it is typically
>>>> a problem with MPI, either an inconsistency with the MPI libraries
>>>> (compiling, linking, and running have to use the same version), or a
>>>> problem with the startup mechanism that MPI uses (there may be a
>>>> timeout). In the latter case, I hope the system administrators will be
>>>> able to help.
>>>>
>>> The run starts and even produces output twice (on the
>>>  32nd iteration), but it then just stalls. I am going to try with the
>>> configuration file you sent, and if that doesn't work, I'll submit a
>>> ticket.
>>>
>>>> -erik
>>>>
>>>> On Fri, Mar 18, 2011 at 11:35 AM, Yosef Zlochower <yosef at astro.rit.edu>
>>>> wrote:
>>>>> Hi,
>>>>>
>>>>>  I am trying some simulations on lonestar using the config file
>>>>> below. I noticed that my simulations hang when I choose large
>>>>> numbers of cores. I haven't yet determined where
>>>>> they hang (the same executable and parfile combination does
>>>>> not hang on fewer cores). Is there a standard config file
>>>>> for lonestar?
>>>>>
>>>>> CPP = cpp
>>>>> FPP = cpp
>>>>> CC  = /opt/apps/intel/11.1/bin/intel64/icc
>>>>> CXX = /opt/apps/intel/11.1/bin/intel64/icpc
>>>>> F77 = /opt/apps/intel/11.1/bin/intel64/ifort
>>>>> F90 = /opt/apps/intel/11.1/bin/intel64/ifort
>>>>>
>>>>> # -inline-debug-info leads to compiler crashes when used with
>>>>> optimisation
>>>>> CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
>>>>> FPPFLAGS = -traditional
>>>>> #CFLAGS   = -g -debug all -traceback -align -std=gnu99 -ansi_alias
>>>>> CFLAGS   = -g -debug all -traceback -align -std=c99 -ansi_alias
>>>>> CXXFLAGS = -g -debug all -traceback -align -restrict
>>>>> F77FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>>>> F90FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>>>>
>>>>> C_LINE_DIRECTIVES = yes
>>>>> F_LINE_DIRECTIVES = yes
>>>>>
>>>>> LDFLAGS =  -Wl,-rpath,/opt/apps/intel11_1/mvapich2/1.6/lib
>>>>> -Wl,-rpath,/opt/apps/
>>>>> intel/11.1/lib/intel64/lib
>>>>>
>>>>> # Include hypre, SPAI, Chaco, spooles, SuperLU_DIST, MUMPS, ParMetis,
>>>>> SCALAPACK,
>>>>>  blacs, g2c, and X11 for PETSc
>>>>> LIBDIRS = /opt/apps/intel/11.1/lib/intel64/lib
>>>>>
>>>>> # -check-uninit aborts for uninitialised variables, which is too strict
>>>>> DEBUG           = no
>>>>> CPP_DEBUG_FLAGS = -DCARPET_DEBUG
>>>>> FPP_DEBUG_FLAGS = -DCARPET_DEBUG
>>>>> C_DEBUG_FLAGS   = -O0 # -check-uninit
>>>>> CXX_DEBUG_FLAGS = -O0 # -check-uninit
>>>>> F77_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>>>> F90_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>>>>
>>>>> # -O3 miscompiles parts of CarpetLib
>>>>> # -ip leads to wrong physics when OpenMP is used
>>>>> OPTIMISE           = yes
>>>>> CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>>>>> FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>>>>> C_OPTIMISE_FLAGS   = -O2 -xW
>>>>> CXX_OPTIMISE_FLAGS = -O2 -xW
>>>>> F77_OPTIMISE_FLAGS = -O2 -xW
>>>>> F90_OPTIMISE_FLAGS = -O2 -xW
>>>>>
>>>>> PROFILE           = no
>>>>> CPP_PROFILE_FLAGS =
>>>>> FPP_PROFILE_FLAGS =
>>>>> C_PROFILE_FLAGS   = -pg
>>>>> CXX_PROFILE_FLAGS = -pg
>>>>> F77_PROFILE_FLAGS = -pg
>>>>> F90_PROFILE_FLAGS = -pg
>>>>>
>>>>> OPENMP           = yes
>>>>> CPP_OPENMP_FLAGS = -openmp
>>>>> FPP_OPENMP_FLAGS = -D_OPENMP
>>>>> C_OPENMP_FLAGS   = -openmp
>>>>> CXX_OPENMP_FLAGS = -openmp
>>>>> F77_OPENMP_FLAGS = -openmp
>>>>> F90_OPENMP_FLAGS = -openmp
>>>>>
>>>>> WARN           = yes
>>>>> CPP_WARN_FLAGS =
>>>>> FPP_WARN_FLAGS =
>>>>> C_WARN_FLAGS   =
>>>>> CXX_WARN_FLAGS =
>>>>> F77_WARN_FLAGS =
>>>>> F90_WARN_FLAGS =
>>>>>
>>>>> BLAS_DIR  =
>>>>>
>>>>> HDF5     = yes
>>>>> HDF5_DIR = BUILD
>>>>>
>>>>> #Previously:
>>>>> #GSL_DIR = /opt/apps/gsl/1.11
>>>>> GSL_DIR = BUILD
>>>>>
>>>>> LAPACK_DIR  =
>>>>> MPI          = CUSTOM
>>>>> MPI_INC_DIRS = /opt/apps/intel11_1/mvapich2/1.6/include/
>>>>> MPI_LIB_DIRS = /opt/apps/intel11_1/mvapich2/1.6/lib/ /opt/ofed/lib64/
>>>>> MPI_LIBS     = mpich rt rdmacm ibverbs ibumad
>>>>>
>>>>> PTHREADS = yes
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dr. Yosef Zlochower
>>>>> Center for Computational Relativity and Gravitation
>>>>> Assistant Professor
>>>>> School of Mathematical Sciences
>>>>> Rochester Institute of Technology
>>>>> 85 Lomb Memorial Drive
>>>>> Rochester, NY 14623
>>>>>
>>>>> Office:74-2067
>>>>> Phone: +1 585-475-6103
>>>>>
>>>>> yosef at astro.rit.edu
>>>>>
>>>>> CONFIDENTIALITY NOTE: The information transmitted, including
>>>>> attachments, is intended only for the person(s) or entity to which it
>>>>> is addressed and may contain confidential and/or privileged material.
>>>>> Any review, retransmission, dissemination or other use of, or taking
>>>>> of any action in reliance upon this information by persons or entities
>>>>> other than the intended recipient is prohibited. If you received this
>>>>> in error, please contact the sender and destroy any copies of this
>>>>> information.
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at einsteintoolkit.org
>>>>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>>>>
>>>>
>>>>
>>>
>>> -