Hi all,
I've been having a great deal of trouble installing ET(K) on our local cluster at Northwestern. While I've been able to install and run the code, I believe something is wrong with how I compiled it with MPI. I had tried to reverse engineer some of the machine/submit files in simfactory/mdb, but the kludged configurations I've tried either stall after Carpet tries to allocate memory or fails in some of the MPI calls. Sometimes it runs correctly, but only on a power of two number of cores.
Could someone who is very familiar with Carpet/MPI and compiling with ICC take a quick look at the configuration I've used and let me know if something is obviously wrong (i.e. incorrect libraries or conflicting compilation flags)? It might be a machine specific problem, but it'd be good to know I've compiled Carpet correctly before moving on to more technical discussions with our sys-admin.
The relevant parts of the cfg (compiled with icc associated with this MPI) are
MPI = OpenMPI OPENMPI_DIR = /software/mpi/openmpi-1.4.3-intel-2011.3 MPI_INC_DIRS = /software/mpi/openmpi-1.4.3-intel-2011.3/include MPI_LIB_DIRS = /software/mpi/openmpi-1.4.3-intel-2011.3/lib /usr/lib64 MPI_LIBS = mpi_cxx mpi rt torque dl pthread PTHREADS = yes
and then submit with
time $OPENMPI_DIR/bin/mpirun -np @NUM_PROCS@ -cpus-per-proc @NUM_THREADS@ @EXECUTABLE@ -L 3 @PARFILE@
Thanks! Carl