[Users] ET_2013_11 run performance

[James Healy]

Hello all,

On Thursday morning, I pulled a fresh checkout of the newest version of 
the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I 
compiled it on stampede using the current stampede.cfg located in 
simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and 
the intel compilers version 13.1.1.163 (enabled through a module load).  
I submitted a short job which I ran previously with ET_2013_05.  The 
results come out the same.  However, the run speed as reported in 
Carpet::physical_time_per_hour is poor. It starts off good, 
approximately the same as with the previous build, but over time drops 
to as low as half the speed over 24 hours of evolution. On recovery from 
checkpoint, the speed is even worse, dropping to below 1/4 of the 
original run speed.

So, I tried using the previous stampede.cfg included in the ET_2013_05 
branch of simfactory, the same one I used to compile my ET_2013_05 
build.  This cfgfile uses the same version of IMPI but different Intel 
compilers (version 13.0.2.146). The run speed shows the same trends as 
when using the newer config file.

The parameter file I am using is EXACTLY the same, not a single change.  
The run is fairly lightweight, not requiring many nodes to run (I used 
2).  It is purely vacuum GR, an R1 configuration (equal-mass, 
non-spinning BBH system with a small initial separation).  Memory usage 
in all three cases is the same, and is stable throughout the run.

Are there new parameters that I should be setting to get similar 
performance as before?  New thorns that I should have active?  If not, 
has anyone else used stampede recently and experienced similar problems?

Thanks,
Jim Healy
Rochester Institute of Technology