Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
The parameter file I am using is EXACTLY the same, not a single change. The run is fairly lightweight, not requiring many nodes to run (I used 2). It is purely vacuum GR, an R1 configuration (equal-mass, non-spinning BBH system with a small initial separation). Memory usage in all three cases is the same, and is stable throughout the run.
Are there new parameters that I should be setting to get similar performance as before? New thorns that I should have active? If not, has anyone else used stampede recently and experienced similar problems?
Thanks, Jim Healy Rochester Institute of Technology