[Users] Carpet/MPI Compilation help

[Carl Rodriguez]

Hi all,

I've been having a great deal of trouble installing ET(K) on our local 
cluster at Northwestern.  While I've been able to install and run the 
code, I believe something is wrong with how I compiled it with MPI.  I 
had tried to reverse engineer some of the machine/submit files in 
simfactory/mdb, but the kludged configurations I've tried either stall 
after Carpet tries to allocate memory or fails in some of the MPI 
calls.  Sometimes it runs correctly, but only on a power of two number 
of cores.

Could someone who is very familiar with Carpet/MPI and compiling with 
ICC take a quick look at the configuration I've used and let me know if 
something is obviously wrong (i.e. incorrect libraries or conflicting 
compilation flags)?  It might be a machine specific problem, but it'd be 
good to know I've compiled Carpet correctly before moving on to more 
technical discussions with our sys-admin.

The relevant parts of the cfg (compiled with icc associated with this 
MPI) are

MPI          = OpenMPI
OPENMPI_DIR = /software/mpi/openmpi-1.4.3-intel-2011.3
MPI_INC_DIRS = /software/mpi/openmpi-1.4.3-intel-2011.3/include
MPI_LIB_DIRS = /software/mpi/openmpi-1.4.3-intel-2011.3/lib /usr/lib64
MPI_LIBS     = mpi_cxx mpi rt torque dl pthread
PTHREADS = yes

and then submit with

time $OPENMPI_DIR/bin/mpirun -np @NUM_PROCS@ -cpus-per-proc 
@NUM_THREADS@ @EXECUTABLE@ -L 3 @PARFILE@

Thanks!
Carl