[Users] config file for lonestar
Ian Hinder
ian.hinder at aei.mpg.de
Fri Mar 18 11:46:12 CDT 2011
On 18 Mar 2011, at 17:30, Erik Schnetter wrote:
> That probably means it's not a startup problem... but it could still
> be something strange with MPI, e.g. during regridding.
>
> Another issue I just remember is running out of memory.
Hi Yosef,
Also, it often helps to use OpenMP, since this reduces the number of MPI processes which need to communicate initially. In your config it is enabled, but are you using it?
>
> -erik
>
> On Fri, Mar 18, 2011 at 12:23 PM, Yosef Zlochower <yosef at astro.rit.edu> wrote:
>> Thanks,
>>
>> Erik Schnetter wrote:
>>
>>> Yosef
>>>
>>> Yes, there is a "standard" config file, but I have not tested it on
>>> large numbers of cores. I attach it below.
>>>
>>> If there is a problem on large numbers of cores, then it is typically
>>> a problem with MPI, either an inconsistency with the MPI libraries
>>> (compiling, linking, and running have to use the same version), or a
>>> problem with the startup mechanism that MPI uses (there may be a
>>> timeout). In the latter case, I hope the system administrators will be
>>> able to help.
>>>
>>
>> The run starts and even produces output twice (on the
>> 32nd iteration), but it then just stalls. I am going to try with the
>> configuration file you sent, and if that doesn't work, I'll submit a
>> ticket.
>>
>>> -erik
>>>
>>> On Fri, Mar 18, 2011 at 11:35 AM, Yosef Zlochower <yosef at astro.rit.edu>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am trying some simulations on lonestar using the config file
>>>> below. I noticed that my simulations hang when I choose large
>>>> numbers of cores. I haven't yet determined where
>>>> they hang (the same executable and parfile combination does
>>>> not hang on fewer cores). Is there a standard config file
>>>> for lonestar?
>>>>
>>>> CPP = cpp
>>>> FPP = cpp
>>>> CC = /opt/apps/intel/11.1/bin/intel64/icc
>>>> CXX = /opt/apps/intel/11.1/bin/intel64/icpc
>>>> F77 = /opt/apps/intel/11.1/bin/intel64/ifort
>>>> F90 = /opt/apps/intel/11.1/bin/intel64/ifort
>>>>
>>>> # -inline-debug-info leads to compiler crashes when used with
>>>> optimisation
>>>> CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
>>>> FPPFLAGS = -traditional
>>>> #CFLAGS = -g -debug all -traceback -align -std=gnu99 -ansi_alias
>>>> CFLAGS = -g -debug all -traceback -align -std=c99 -ansi_alias
>>>> CXXFLAGS = -g -debug all -traceback -align -restrict
>>>> F77FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>>> F90FLAGS = -g -debug all -traceback -align -pad -w95 -cm
>>>>
>>>> C_LINE_DIRECTIVES = yes
>>>> F_LINE_DIRECTIVES = yes
>>>>
>>>> LDFLAGS = -Wl,-rpath,/opt/apps/intel11_1/mvapich2/1.6/lib
>>>> -Wl,-rpath,/opt/apps/
>>>> intel/11.1/lib/intel64/lib
>>>>
>>>> # Include hypre, SPAI, Chaco, spooles, SuperLU_DIST, MUMPS, ParMetis,
>>>> SCALAPACK,
>>>> blacs, g2c, and X11 for PETSc
>>>> LIBDIRS = /opt/apps/intel/11.1/lib/intel64/lib
>>>>
>>>> # -check-uninit aborts for uninitialised variables, which is too strict
>>>> DEBUG = no
>>>> CPP_DEBUG_FLAGS = -DCARPET_DEBUG
>>>> FPP_DEBUG_FLAGS = -DCARPET_DEBUG
>>>> C_DEBUG_FLAGS = -O0 # -check-uninit
>>>> CXX_DEBUG_FLAGS = -O0 # -check-uninit
>>>> F77_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>>> F90_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
>>>>
>>>> # -O3 miscompiles parts of CarpetLib
>>>> # -ip leads to wrong physics when OpenMP is used
>>>> OPTIMISE = yes
>>>> CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>>>> FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
>>>> C_OPTIMISE_FLAGS = -O2 -xW
>>>> CXX_OPTIMISE_FLAGS = -O2 -xW
>>>> F77_OPTIMISE_FLAGS = -O2 -xW
>>>> F90_OPTIMISE_FLAGS = -O2 -xW
>>>>
>>>> PROFILE = no
>>>> CPP_PROFILE_FLAGS =
>>>> FPP_PROFILE_FLAGS =
>>>> C_PROFILE_FLAGS = -pg
>>>> CXX_PROFILE_FLAGS = -pg
>>>> F77_PROFILE_FLAGS = -pg
>>>> F90_PROFILE_FLAGS = -pg
>>>>
>>>> OPENMP = yes
>>>> CPP_OPENMP_FLAGS = -openmp
>>>> FPP_OPENMP_FLAGS = -D_OPENMP
>>>> C_OPENMP_FLAGS = -openmp
>>>> CXX_OPENMP_FLAGS = -openmp
>>>> F77_OPENMP_FLAGS = -openmp
>>>> F90_OPENMP_FLAGS = -openmp
>>>>
>>>> WARN = yes
>>>> CPP_WARN_FLAGS =
>>>> FPP_WARN_FLAGS =
>>>> C_WARN_FLAGS =
>>>> CXX_WARN_FLAGS =
>>>> F77_WARN_FLAGS =
>>>> F90_WARN_FLAGS =
>>>>
>>>> BLAS_DIR =
>>>>
>>>> HDF5 = yes
>>>> HDF5_DIR = BUILD
>>>>
>>>> #Previously:
>>>> #GSL_DIR = /opt/apps/gsl/1.11
>>>> GSL_DIR = BUILD
>>>>
>>>> LAPACK_DIR =
>>>> MPI = CUSTOM
>>>> MPI_INC_DIRS = /opt/apps/intel11_1/mvapich2/1.6/include/
>>>> MPI_LIB_DIRS = /opt/apps/intel11_1/mvapich2/1.6/lib/ /opt/ofed/lib64/
>>>> MPI_LIBS = mpich rt rdmacm ibverbs ibumad
>>>>
>>>> PTHREADS = yes
>>>>
>>>>
>>>>
>>>> --
>>>> Dr. Yosef Zlochower
>>>> Center for Computational Relativity and Gravitation
>>>> Assistant Professor
>>>> School of Mathematical Sciences
>>>> Rochester Institute of Technology
>>>> 85 Lomb Memorial Drive
>>>> Rochester, NY 14623
>>>>
>>>> Office:74-2067
>>>> Phone: +1 585-475-6103
>>>>
>>>> yosef at astro.rit.edu
>>>>
>>>> CONFIDENTIALITY NOTE: The information transmitted, including
>>>> attachments, is intended only for the person(s) or entity to which it
>>>> is addressed and may contain confidential and/or privileged material.
>>>> Any review, retransmission, dissemination or other use of, or taking
>>>> of any action in reliance upon this information by persons or entities
>>>> other than the intended recipient is prohibited. If you received this
>>>> in error, please contact the sender and destroy any copies of this
>>>> information.
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at einsteintoolkit.org
>>>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Dr. Yosef Zlochower
>> Center for Computational Relativity and Gravitation
>> Assistant Professor
>> School of Mathematical Sciences
>> Rochester Institute of Technology
>> 85 Lomb Memorial Drive
>> Rochester, NY 14623
>>
>> Office:74-2067
>> Phone: +1 585-475-6103
>>
>> yosef at astro.rit.edu
>>
>> CONFIDENTIALITY NOTE: The information transmitted, including
>> attachments, is intended only for the person(s) or entity to which it
>> is addressed and may contain confidential and/or privileged material.
>> Any review, retransmission, dissemination or other use of, or taking
>> of any action in reliance upon this information by persons or entities
>> other than the intended recipient is prohibited. If you received this
>> in error, please contact the sender and destroy any copies of this
>> information.
>>
>
>
>
> --
> Erik Schnetter <schnetter at cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Ian Hinder
ian.hinder at aei.mpg.de
More information about the Users
mailing list