Thanks,
Erik Schnetter wrote:
Yosef
Yes, there is a "standard" config file, but I have not tested it on large numbers of cores. I attach it below.
If there is a problem on large numbers of cores, then it is typically a problem with MPI, either an inconsistency with the MPI libraries (compiling, linking, and running have to use the same version), or a problem with the startup mechanism that MPI uses (there may be a timeout). In the latter case, I hope the system administrators will be able to help.
The run starts and even produces output twice (on the 32nd iteration), but it then just stalls. I am going to try with the configuration file you sent, and if that doesn't work, I'll submit a ticket.
-erik
On Fri, Mar 18, 2011 at 11:35 AM, Yosef Zlochower yosef@astro.rit.edu wrote:
Hi,
I am trying some simulations on lonestar using the config file below. I noticed that my simulations hang when I choose large numbers of cores. I haven't yet determined where they hang (the same executable and parfile combination does not hang on fewer cores). Is there a standard config file for lonestar?
CPP = cpp FPP = cpp CC = /opt/apps/intel/11.1/bin/intel64/icc CXX = /opt/apps/intel/11.1/bin/intel64/icpc F77 = /opt/apps/intel/11.1/bin/intel64/ifort F90 = /opt/apps/intel/11.1/bin/intel64/ifort
# -inline-debug-info leads to compiler crashes when used with optimisation CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK FPPFLAGS = -traditional #CFLAGS = -g -debug all -traceback -align -std=gnu99 -ansi_alias CFLAGS = -g -debug all -traceback -align -std=c99 -ansi_alias CXXFLAGS = -g -debug all -traceback -align -restrict F77FLAGS = -g -debug all -traceback -align -pad -w95 -cm F90FLAGS = -g -debug all -traceback -align -pad -w95 -cm
C_LINE_DIRECTIVES = yes F_LINE_DIRECTIVES = yes
LDFLAGS = -Wl,-rpath,/opt/apps/intel11_1/mvapich2/1.6/lib -Wl,-rpath,/opt/apps/ intel/11.1/lib/intel64/lib
# Include hypre, SPAI, Chaco, spooles, SuperLU_DIST, MUMPS, ParMetis, SCALAPACK, blacs, g2c, and X11 for PETSc LIBDIRS = /opt/apps/intel/11.1/lib/intel64/lib
# -check-uninit aborts for uninitialised variables, which is too strict DEBUG = no CPP_DEBUG_FLAGS = -DCARPET_DEBUG FPP_DEBUG_FLAGS = -DCARPET_DEBUG C_DEBUG_FLAGS = -O0 # -check-uninit CXX_DEBUG_FLAGS = -O0 # -check-uninit F77_DEBUG_FLAGS = -O0 -check bounds -check format # -check all F90_DEBUG_FLAGS = -O0 -check bounds -check format # -check all
# -O3 miscompiles parts of CarpetLib # -ip leads to wrong physics when OpenMP is used OPTIMISE = yes CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG C_OPTIMISE_FLAGS = -O2 -xW CXX_OPTIMISE_FLAGS = -O2 -xW F77_OPTIMISE_FLAGS = -O2 -xW F90_OPTIMISE_FLAGS = -O2 -xW
PROFILE = no CPP_PROFILE_FLAGS = FPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F77_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg
OPENMP = yes CPP_OPENMP_FLAGS = -openmp FPP_OPENMP_FLAGS = -D_OPENMP C_OPENMP_FLAGS = -openmp CXX_OPENMP_FLAGS = -openmp F77_OPENMP_FLAGS = -openmp F90_OPENMP_FLAGS = -openmp
WARN = yes CPP_WARN_FLAGS = FPP_WARN_FLAGS = C_WARN_FLAGS = CXX_WARN_FLAGS = F77_WARN_FLAGS = F90_WARN_FLAGS =
BLAS_DIR =
HDF5 = yes HDF5_DIR = BUILD
#Previously: #GSL_DIR = /opt/apps/gsl/1.11 GSL_DIR = BUILD
LAPACK_DIR = MPI = CUSTOM MPI_INC_DIRS = /opt/apps/intel11_1/mvapich2/1.6/include/ MPI_LIB_DIRS = /opt/apps/intel11_1/mvapich2/1.6/lib/ /opt/ofed/lib64/ MPI_LIBS = mpich rt rdmacm ibverbs ibumad
PTHREADS = yes
-- Dr. Yosef Zlochower Center for Computational Relativity and Gravitation Assistant Professor School of Mathematical Sciences Rochester Institute of Technology 85 Lomb Memorial Drive Rochester, NY 14623
Office:74-2067 Phone: +1 585-475-6103
yosef@astro.rit.edu
CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users