Hello Jim,
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Have you tried on a different machine as well?
The parameter file I am using is EXACTLY the same, not a single change. The run is fairly lightweight, not requiring many nodes to run (I used 2). It is purely vacuum GR, an R1 configuration (equal-mass, non-spinning BBH system with a small initial separation). Memory usage in all three cases is the same, and is stable throughout the run.
Are there new parameters that I should be setting to get similar performance as before? New thorns that I should have active? If not, has anyone else used stampede recently and experienced similar problems?
In particular, please have a look at ticket #1512 https://trac.einsteintoolkit.org/ticket/1512 which contains a small parfile which you can use to test for particular issue in loopcontrol. The test runs in ~2 minutes or so and you should be able to verify if you see the slowdown you experienced. A workaround is to set LoopControl::settle_after_iteration = 0 . The slowdown after recovery however does not fit this picture.
Yours, Roland