[Users] config file for lonestar

[Yosef Zlochower]

Thanks,

Erik Schnetter wrote:

> Yosef
> 
> Yes, there is a "standard" config file, but I have not tested it on
> large numbers of cores. I attach it below.
> 
> If there is a problem on large numbers of cores, then it is typically
> a problem with MPI, either an inconsistency with the MPI libraries
> (compiling, linking, and running have to use the same version), or a
> problem with the startup mechanism that MPI uses (there may be a
> timeout). In the latter case, I hope the system administrators will be
> able to help.
> 

The run starts and even produces output twice (on the
  32nd iteration), but it then just stalls. I am going to try with the
configuration file you sent, and if that doesn't work, I'll submit a
ticket.

> -erik
> 
> On Fri, Mar 18, 2011 at 11:35 AM, Yosef Zlochower <yosef at astro.rit.edu> wrote:
>> Hi,
>>
>>  I am trying some simulations on lonestar using the config file
>> below. I noticed that my simulations hang when I choose large
>> numbers of cores. I haven't yet determined where
>> they hang (the same executable and parfile combination does
>> not hang on fewer cores). Is there a standard config file
>> for lonestar?
>>
>> CPP = cpp
>> FPP = cpp
>> CC  = /opt/apps/intel/11.1/bin/intel64/icc
>> CXX = /opt/apps/intel/11.1/bin/intel64/icpc
>> F77 = /opt/apps/intel/11.1/bin/intel64/ifort
>> F90 = /opt/apps/intel/11.1/bin/intel64/ifort
>>
>> # -inline-debug-info leads to compiler crashes when used with optimisation
>> CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
>> FPPFLAGS = -traditional
>> #CFLAGS   = -g -debug all -traceback -align -std=gnu99 -ansi_alias
>> CFLAGS   = -g -debug all -traceback -align -std=c99 -ansi_alias
>> CXXFLAGS = -g -debug all -traceback -align -restrict
>> F77FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>> F90FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>
>> C_LINE_DIRECTIVES = yes
>> F_LINE_DIRECTIVES = yes
>>
>> LDFLAGS =  -Wl,-rpath,/opt/apps/intel11_1/mvapich2/1.6/lib
>> -Wl,-rpath,/opt/apps/
>> intel/11.1/lib/intel64/lib
>>
>> # Include hypre, SPAI, Chaco, spooles, SuperLU_DIST, MUMPS, ParMetis,
>> SCALAPACK,
>>  blacs, g2c, and X11 for PETSc
>> LIBDIRS = /opt/apps/intel/11.1/lib/intel64/lib
>>
>> # -check-uninit aborts for uninitialised variables, which is too strict
>> DEBUG           = no
>> CPP_DEBUG_FLAGS = -DCARPET_DEBUG
>> FPP_DEBUG_FLAGS = -DCARPET_DEBUG
>> C_DEBUG_FLAGS   = -O0 # -check-uninit
>> CXX_DEBUG_FLAGS = -O0 # -check-uninit
>> F77_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>> F90_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>
>> # -O3 miscompiles parts of CarpetLib
>> # -ip leads to wrong physics when OpenMP is used
>> OPTIMISE           = yes
>> CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>> FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>> C_OPTIMISE_FLAGS   = -O2 -xW
>> CXX_OPTIMISE_FLAGS = -O2 -xW
>> F77_OPTIMISE_FLAGS = -O2 -xW
>> F90_OPTIMISE_FLAGS = -O2 -xW
>>
>> PROFILE           = no
>> CPP_PROFILE_FLAGS =
>> FPP_PROFILE_FLAGS =
>> C_PROFILE_FLAGS   = -pg
>> CXX_PROFILE_FLAGS = -pg
>> F77_PROFILE_FLAGS = -pg
>> F90_PROFILE_FLAGS = -pg
>>
>> OPENMP           = yes
>> CPP_OPENMP_FLAGS = -openmp
>> FPP_OPENMP_FLAGS = -D_OPENMP
>> C_OPENMP_FLAGS   = -openmp
>> CXX_OPENMP_FLAGS = -openmp
>> F77_OPENMP_FLAGS = -openmp
>> F90_OPENMP_FLAGS = -openmp
>>
>> WARN           = yes
>> CPP_WARN_FLAGS =
>> FPP_WARN_FLAGS =
>> C_WARN_FLAGS   =
>> CXX_WARN_FLAGS =
>> F77_WARN_FLAGS =
>> F90_WARN_FLAGS =
>>
>> BLAS_DIR  =
>>
>> HDF5     = yes
>> HDF5_DIR = BUILD
>>
>> #Previously:
>> #GSL_DIR = /opt/apps/gsl/1.11
>> GSL_DIR = BUILD
>>
>> LAPACK_DIR  =
>> MPI          = CUSTOM
>> MPI_INC_DIRS = /opt/apps/intel11_1/mvapich2/1.6/include/
>> MPI_LIB_DIRS = /opt/apps/intel11_1/mvapich2/1.6/lib/ /opt/ofed/lib64/
>> MPI_LIBS     = mpich rt rdmacm ibverbs ibumad
>>
>> PTHREADS = yes
>>
>>
>>
>> --
>> Dr. Yosef Zlochower
>> Center for Computational Relativity and Gravitation
>> Assistant Professor
>> School of Mathematical Sciences
>> Rochester Institute of Technology
>> 85 Lomb Memorial Drive
>> Rochester, NY 14623
>>
>> Office:74-2067
>> Phone: +1 585-475-6103
>>
>> yosef at astro.rit.edu
>>
>> CONFIDENTIALITY NOTE: The information transmitted, including
>> attachments, is intended only for the person(s) or entity to which it
>> is addressed and may contain confidential and/or privileged material.
>> Any review, retransmission, dissemination or other use of, or taking
>> of any action in reliance upon this information by persons or entities
>> other than the intended recipient is prohibited. If you received this
>> in error, please contact the sender and destroy any copies of this
>> information.
>> _______________________________________________
>> Users mailing list
>> Users at einsteintoolkit.org
>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>
> 
> 
> 


-- 
Dr. Yosef Zlochower
Center for Computational Relativity and Gravitation
Assistant Professor
School of Mathematical Sciences
Rochester Institute of Technology
85 Lomb Memorial Drive
Rochester, NY 14623

Office:74-2067
Phone: +1 585-475-6103

yosef at astro.rit.edu

CONFIDENTIALITY NOTE: The information transmitted, including
attachments, is intended only for the person(s) or entity to which it
is addressed and may contain confidential and/or privileged material.
Any review, retransmission, dissemination or other use of, or taking
of any action in reliance upon this information by persons or entities
other than the intended recipient is prohibited. If you received this
in error, please contact the sender and destroy any copies of this
information.