[Users] config file for lonestar

Fri Mar 18 11:30:51 CDT 2011

That probably means it's not a startup problem... but it could still
be something strange with MPI, e.g. during regridding.

Another issue I just remember is running out of memory.

-erik

On Fri, Mar 18, 2011 at 12:23 PM, Yosef Zlochower <yosef at astro.rit.edu> wrote:
> Thanks,
>
> Erik Schnetter wrote:
>
>> Yosef
>>
>> Yes, there is a "standard" config file, but I have not tested it on
>> large numbers of cores. I attach it below.
>>
>> If there is a problem on large numbers of cores, then it is typically
>> a problem with MPI, either an inconsistency with the MPI libraries
>> (compiling, linking, and running have to use the same version), or a
>> problem with the startup mechanism that MPI uses (there may be a
>> timeout). In the latter case, I hope the system administrators will be
>> able to help.
>>
>
> The run starts and even produces output twice (on the
>  32nd iteration), but it then just stalls. I am going to try with the
> configuration file you sent, and if that doesn't work, I'll submit a
> ticket.
>
>> -erik
>>
>> On Fri, Mar 18, 2011 at 11:35 AM, Yosef Zlochower <yosef at astro.rit.edu>
>> wrote:
>>>
>>> Hi,
>>>
>>>  I am trying some simulations on lonestar using the config file
>>> below. I noticed that my simulations hang when I choose large
>>> numbers of cores. I haven't yet determined where
>>> they hang (the same executable and parfile combination does
>>> not hang on fewer cores). Is there a standard config file
>>> for lonestar?
>>>
>>> CPP = cpp
>>> FPP = cpp
>>> CC  = /opt/apps/intel/11.1/bin/intel64/icc
>>> CXX = /opt/apps/intel/11.1/bin/intel64/icpc
>>> F77 = /opt/apps/intel/11.1/bin/intel64/ifort
>>> F90 = /opt/apps/intel/11.1/bin/intel64/ifort
>>>
>>> # -inline-debug-info leads to compiler crashes when used with
>>> optimisation
>>> CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
>>> FPPFLAGS = -traditional
>>> #CFLAGS   = -g -debug all -traceback -align -std=gnu99 -ansi_alias
>>> CFLAGS   = -g -debug all -traceback -align -std=c99 -ansi_alias
>>> CXXFLAGS = -g -debug all -traceback -align -restrict
>>> F77FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>> F90FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>>
>>> C_LINE_DIRECTIVES = yes
>>> F_LINE_DIRECTIVES = yes
>>>
>>> LDFLAGS =  -Wl,-rpath,/opt/apps/intel11_1/mvapich2/1.6/lib
>>> -Wl,-rpath,/opt/apps/
>>> intel/11.1/lib/intel64/lib
>>>
>>> # Include hypre, SPAI, Chaco, spooles, SuperLU_DIST, MUMPS, ParMetis,
>>> SCALAPACK,
>>>  blacs, g2c, and X11 for PETSc
>>> LIBDIRS = /opt/apps/intel/11.1/lib/intel64/lib
>>>
>>> # -check-uninit aborts for uninitialised variables, which is too strict
>>> DEBUG           = no
>>> CPP_DEBUG_FLAGS = -DCARPET_DEBUG
>>> FPP_DEBUG_FLAGS = -DCARPET_DEBUG
>>> C_DEBUG_FLAGS   = -O0 # -check-uninit
>>> CXX_DEBUG_FLAGS = -O0 # -check-uninit
>>> F77_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>> F90_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>>
>>> # -O3 miscompiles parts of CarpetLib
>>> # -ip leads to wrong physics when OpenMP is used
>>> OPTIMISE           = yes
>>> CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>>> FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>>> C_OPTIMISE_FLAGS   = -O2 -xW
>>> CXX_OPTIMISE_FLAGS = -O2 -xW
>>> F77_OPTIMISE_FLAGS = -O2 -xW
>>> F90_OPTIMISE_FLAGS = -O2 -xW
>>>
>>> PROFILE           = no
>>> CPP_PROFILE_FLAGS =
>>> FPP_PROFILE_FLAGS =
>>> C_PROFILE_FLAGS   = -pg
>>> CXX_PROFILE_FLAGS = -pg
>>> F77_PROFILE_FLAGS = -pg
>>> F90_PROFILE_FLAGS = -pg
>>>
>>> OPENMP           = yes
>>> CPP_OPENMP_FLAGS = -openmp
>>> FPP_OPENMP_FLAGS = -D_OPENMP
>>> C_OPENMP_FLAGS   = -openmp
>>> CXX_OPENMP_FLAGS = -openmp
>>> F77_OPENMP_FLAGS = -openmp
>>> F90_OPENMP_FLAGS = -openmp
>>>
>>> WARN           = yes
>>> CPP_WARN_FLAGS =
>>> FPP_WARN_FLAGS =
>>> C_WARN_FLAGS   =
>>> CXX_WARN_FLAGS =
>>> F77_WARN_FLAGS =
>>> F90_WARN_FLAGS =
>>>
>>> BLAS_DIR  =
>>>
>>> HDF5     = yes
>>> HDF5_DIR = BUILD
>>>
>>> #Previously:
>>> #GSL_DIR = /opt/apps/gsl/1.11
>>> GSL_DIR = BUILD
>>>
>>> LAPACK_DIR  =
>>> MPI          = CUSTOM
>>> MPI_INC_DIRS = /opt/apps/intel11_1/mvapich2/1.6/include/
>>> MPI_LIB_DIRS = /opt/apps/intel11_1/mvapich2/1.6/lib/ /opt/ofed/lib64/
>>> MPI_LIBS     = mpich rt rdmacm ibverbs ibumad
>>>
>>> PTHREADS = yes
>>>
>>>
>>>
>>> --
>>> Dr. Yosef Zlochower
>>> Center for Computational Relativity and Gravitation
>>> Assistant Professor
>>> School of Mathematical Sciences
>>> Rochester Institute of Technology
>>> 85 Lomb Memorial Drive
>>> Rochester, NY 14623
>>>
>>> Office:74-2067
>>> Phone: +1 585-475-6103
>>>
>>> yosef at astro.rit.edu
>>>
>>> CONFIDENTIALITY NOTE: The information transmitted, including
>>> attachments, is intended only for the person(s) or entity to which it
>>> is addressed and may contain confidential and/or privileged material.
>>> Any review, retransmission, dissemination or other use of, or taking
>>> of any action in reliance upon this information by persons or entities
>>> other than the intended recipient is prohibited. If you received this
>>> in error, please contact the sender and destroy any copies of this
>>> information.
>>> _______________________________________________
>>> Users mailing list
>>> Users at einsteintoolkit.org
>>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>>
>>
>>
>>
>
>
> --
> Dr. Yosef Zlochower
> Center for Computational Relativity and Gravitation
> Assistant Professor
> School of Mathematical Sciences
> Rochester Institute of Technology
> 85 Lomb Memorial Drive
> Rochester, NY 14623
>
> Office:74-2067
> Phone: +1 585-475-6103
>
> yosef at astro.rit.edu
>
> CONFIDENTIALITY NOTE: The information transmitted, including
> attachments, is intended only for the person(s) or entity to which it
> is addressed and may contain confidential and/or privileged material.
> Any review, retransmission, dissemination or other use of, or taking
> of any action in reliance upon this information by persons or entities
> other than the intended recipient is prohibited. If you received this
> in error, please contact the sender and destroy any copies of this
> information.
>

-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/