Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
The parameter file I am using is EXACTLY the same, not a single change. The run is fairly lightweight, not requiring many nodes to run (I used 2). It is purely vacuum GR, an R1 configuration (equal-mass, non-spinning BBH system with a small initial separation). Memory usage in all three cases is the same, and is stable throughout the run.
Are there new parameters that I should be setting to get similar performance as before? New thorns that I should have active? If not, has anyone else used stampede recently and experienced similar problems?
Thanks, Jim Healy Rochester Institute of Technology
Hello Jim,
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Have you tried on a different machine as well?
The parameter file I am using is EXACTLY the same, not a single change. The run is fairly lightweight, not requiring many nodes to run (I used 2). It is purely vacuum GR, an R1 configuration (equal-mass, non-spinning BBH system with a small initial separation). Memory usage in all three cases is the same, and is stable throughout the run.
Are there new parameters that I should be setting to get similar performance as before? New thorns that I should have active? If not, has anyone else used stampede recently and experienced similar problems?
In particular, please have a look at ticket #1512 https://trac.einsteintoolkit.org/ticket/1512 which contains a small parfile which you can use to test for particular issue in loopcontrol. The test runs in ~2 minutes or so and you should be able to verify if you see the slowdown you experienced. A workaround is to set LoopControl::settle_after_iteration = 0 . The slowdown after recovery however does not fit this picture.
Yours, Roland
Just a quick follow-up to my run performance problem.
The culprit was the issue with LoopControl. Setting the parameter "LoopControl::settle_after_iteration = 0" reproduced the previous run performance with a performance boost of ~2% over the ET_2013_05 version. I also tried using MVAPICH2 instead of Intel MPI and saw the same type of behavior when using/not using the above parameter.
Thanks for your help, Jim Healy
On 01/20/2014 01:07 AM, Roland Haas wrote:
Hello Jim,
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Have you tried on a different machine as well?
The parameter file I am using is EXACTLY the same, not a single change. The run is fairly lightweight, not requiring many nodes to run (I used 2). It is purely vacuum GR, an R1 configuration (equal-mass, non-spinning BBH system with a small initial separation). Memory usage in all three cases is the same, and is stable throughout the run.
Are there new parameters that I should be setting to get similar performance as before? New thorns that I should have active? If not, has anyone else used stampede recently and experienced similar problems?
In particular, please have a look at ticket #1512 https://trac.einsteintoolkit.org/ticket/1512 which contains a small parfile which you can use to test for particular issue in loopcontrol. The test runs in ~2 minutes or so and you should be able to verify if you see the slowdown you experienced. A workaround is to set LoopControl::settle_after_iteration = 0 . The slowdown after recovery however does not fit this picture.
Yours, Roland
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 20 Jan 2014, at 06:14, James Healy jchsma@rit.edu wrote:
Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release - You don't get the slowdown with the ET_2013_05 release - You do get the slowdown if you use the current ET_2013_11 release with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
We didn't change the MPI version before the release, as that would have been quite an invasive change at that point. However, I would consider backporting this, after suitable discussion.
Of course, your problem might be unrelated to the version of MPI. I am running perfectly fine on stampede with the current trunk (MVAPICH2); runs have a consistent speed and retain this speed after recovery.
On 01/20/2014 08:06 AM, Ian Hinder wrote:
On 20 Jan 2014, at 06:14, James Healy jchsma@rit.edu wrote:
Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release
- You don't get the slowdown with the ET_2013_05 release
- You do get the slowdown if you use the current ET_2013_11 release with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node do your production runs typically use?
We didn't change the MPI version before the release, as that would have been quite an invasive change at that point. However, I would consider backporting this, after suitable discussion.
Of course, your problem might be unrelated to the version of MPI. I am running perfectly fine on stampede with the current trunk (MVAPICH2); runs have a consistent speed and retain this speed after recovery.
On 20 Jan 2014, at 14:23, Yosef Zlochower yosef@astro.rit.edu wrote:
On 01/20/2014 08:06 AM, Ian Hinder wrote:
On 20 Jan 2014, at 06:14, James Healy jchsma@rit.edu wrote:
Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release
- You don't get the slowdown with the ET_2013_05 release
- You do get the slowdown if you use the current ET_2013_11 release with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node do your production runs typically use?
I'm using exactly the default simfactory config from the current trunk, so you can see the modules etc there. Checkpointing (and recovery works fine). I usually aim for something like 75% memory usage for production runs.
We didn't change the MPI version before the release, as that would have been quite an invasive change at that point. However, I would consider backporting this, after suitable discussion.
Of course, your problem might be unrelated to the version of MPI. I am running perfectly fine on stampede with the current trunk (MVAPICH2); runs have a consistent speed and retain this speed after recovery.
On 1/20/14 10:26 PM, Ian Hinder wrote:
On 20 Jan 2014, at 14:23, Yosef Zlochower <yosef@astro.rit.edu mailto:yosef@astro.rit.edu> wrote:
On 01/20/2014 08:06 AM, Ian Hinder wrote:
On 20 Jan 2014, at 06:14, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:
Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release
- You don't get the slowdown with the ET_2013_05 release
- You do get the slowdown if you use the current ET_2013_11 release
with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node do your production runs typically use?
I'm using exactly the default simfactory config from the current trunk, so you can see the modules etc there. Checkpointing (and recovery works fine). I usually aim for something like 75% memory usage for production runs.
Hello, I would like to report a different problem with the simfactory settings for stampede: with ET Noether or trunk the job start/end emails are not sent (or at least they do not reach the Osaka University server; I had the systems administrators check). I receive the emails if I use the simfactory of Gauss. In particular, if I copy just the stampede.ini from Gauss to Noether (and no other files) and recompile, I do receive the emails.
Luca
Luca
The emails depend on the settings in the submit script, i.e. the file simfactory/mdb/submitscripts/stampede.sub. The file "stampede.ini" should not matter.
-erik
On Jan 22, 2014, at 20:41 , Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:
On 1/20/14 10:26 PM, Ian Hinder wrote:
On 20 Jan 2014, at 14:23, Yosef Zlochower <yosef@astro.rit.edu mailto:yosef@astro.rit.edu> wrote:
On 01/20/2014 08:06 AM, Ian Hinder wrote:
On 20 Jan 2014, at 06:14, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:
Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release
- You don't get the slowdown with the ET_2013_05 release
- You do get the slowdown if you use the current ET_2013_11 release
with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node do your production runs typically use?
I'm using exactly the default simfactory config from the current trunk, so you can see the modules etc there. Checkpointing (and recovery works fine). I usually aim for something like 75% memory usage for production runs.
Hello, I would like to report a different problem with the simfactory settings for stampede: with ET Noether or trunk the job start/end emails are not sent (or at least they do not reach the Osaka University server; I had the systems administrators check). I receive the emails if I use the simfactory of Gauss. In particular, if I copy just the stampede.ini from Gauss to Noether (and no other files) and recompile, I do receive the emails.
Luca
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 23 Jan 2014, at 02:51, Erik Schnetter schnetter@cct.lsu.edu wrote:
Luca
The emails depend on the settings in the submit script, i.e. the file simfactory/mdb/submitscripts/stampede.sub. The file "stampede.ini" should not matter.
I just checked, and there is nothing different between the stampede.ini files for the two versions that should affect whether the job emails are sent. There is also no change at all between the submit scripts between the two versions.
Assuming that you have two simulations, one where the emails are sent, and one where they are not, can you perform a diff between the recorded submission script in the simulation SIMFACTORY directory between the two simulations and post the output to the list?
If you are using the versions from the repository, you should get:
--- a/mdb/machines/stampede.ini +++ b/mdb/machines/stampede.ini @@ -1,6 +1,6 @@ [stampede]
-# last-tested-on: 2013-04-29 +# last-tested-on: 2013-11-04 # last-tested-by: Erik Schnetter schnetter@gmail.com
# NOTE: This machine configuration uses only the regular CPUs of @@ -17,7 +17,7 @@ status = experimental # Access to this machine hostname = stampede.tacc.utexas.edu rsynccmd = /home1/00507/eschnett/rsync-3.0.9/bin/rsync -envsetup = module unload mvapich2 && module load impi +envsetup = module load intel/13.1.1.163 && module unload mvapich2 && module load impi/4.1.1.036 && module load papi aliaspattern = ^login[1234](.stampede.tacc.utexas.edu)?$
# Source tree management @@ -40,6 +40,16 @@ disabled-thorns = <<EOT LSUDevelopment/WaveToyNoGhostsPETSc TAT/TATPETSc EOT +enabled-thorns = <<EOT +# CactusTest/TestAllTypes +# ExternalLibraries/OpenCL +# CactusExamples/WaveToyOpenCL +# CactusUtils/Accelerator +# CactusUtils/OpenCLRunTime +# McLachlan/ML_BSSN_CL +# McLachlan/ML_BSSN_CL_Helper +# McLachlan/ML_WaveToy_CL +EOT optionlist = stampede.cfg submitscript = stampede.sub runscript = stampede.run @@ -76,15 +86,15 @@ nodes = 6400 min-ppn = 16 allocation = NO_ALLOCATION queue = normal # [normal, large, development] -maxwalltime = 24:00:00 # development has 4:0:0 +maxwalltime = 48:00:00 # development has 4:0:0 maxqueueslots = 49 -submit = sbatch @SCRIPTFILE@ +submit = sbatch @SCRIPTFILE@; sleep 60 getstatus = squeue -j @JOB_ID@ stop = scancel @JOB_ID@ submitpattern = Submitted batch job ([0-9]+) -statuspattern = ' @JOB_ID@ ' +statuspattern = '@JOB_ID@ ' queuedpattern = ' PD ' -runningpattern = ' R ' +runningpattern = ' (CF|CG|R|TO) ' holdingpattern = ' S ' #exechost = head -n 1 SIMFACTORY/NODES #exechostpattern = ^(\S+)
Since you say that changing the stampede.ini file causes the emails to appear, please can you also post the diff between the two stampede.ini files that you are using?
Maybe there is some weird interaction between the queuing system and some environment settings in stampede.ini, e.g. an environment variable set by the modules.
I never receive email notifications from stampede, even though I am using the default submission script. I assumed it was just broken.
-erik
On Jan 22, 2014, at 20:41 , Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:
On 1/20/14 10:26 PM, Ian Hinder wrote:
On 20 Jan 2014, at 14:23, Yosef Zlochower <yosef@astro.rit.edu mailto:yosef@astro.rit.edu> wrote:
On 01/20/2014 08:06 AM, Ian Hinder wrote:
On 20 Jan 2014, at 06:14, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:
Hello all,
On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed.
So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file.
Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release
- You don't get the slowdown with the ET_2013_05 release
- You do get the slowdown if you use the current ET_2013_11 release
with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node do your production runs typically use?
I'm using exactly the default simfactory config from the current trunk, so you can see the modules etc there. Checkpointing (and recovery works fine). I usually aim for something like 75% memory usage for production runs.
Hello, I would like to report a different problem with the simfactory settings for stampede: with ET Noether or trunk the job start/end emails are not sent (or at least they do not reach the Osaka University server; I had the systems administrators check). I receive the emails if I use the simfactory of Gauss. In particular, if I copy just the stampede.ini from Gauss to Noether (and no other files) and recompile, I do receive the emails.
Luca
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu/.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello.
OK, so the emails not being sent is a known problem.
Here is the diff between my Gauss simfactory (emails working) and my Noether simfactory (emails not working). Both are repository versions.
~> diff -ur Cactus_Noether/simfactory Cactus_Gauss/simfactory/ Only in Cactus_Gauss/simfactory/.svn/pristine/00: 00d3f989d1b1a9928c40c7f3d4878c6cd40e0da6.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine: 05 Only in Cactus_Gauss/simfactory/.svn/pristine/06: 0686da6a0b024c3b7a1c190cb357ce266d80b8de.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine/24: 24336cbf8816e71d47912c56a5fbb0d9d3e002a9.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine/2b: 2ba3fdbaef423054e91d12657a045899fcdbcd87.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine/45: 456b5848c3702e015f203c6ba1dd8e4e044a91c7.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine/48: 48833b1078909902047c656db2c7d538ace7b84f.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine: 50 Only in Cactus_Gauss/simfactory/.svn/pristine: 5e Only in Cactus_Gauss/simfactory/.svn/pristine: 67 Only in Cactus_Gauss/simfactory/.svn/pristine/69: 69616b60ff0814f4ac227c5b37a5c40eeaff688f.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine/6f: 6f80c656a7e0f597ffdd8d826af35361ab0bf116.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine/84: 8461ad8dc9d4a64d707fa9bb4cf5f4e544a65f41.svn-base Only in Cactus_Gauss/simfactory/.svn/pristine: b6 Binary files Cactus_Noether/simfactory/.svn/wc.db and Cactus_Gauss/simfactory/.svn/wc.db differ diff -ur Cactus_Noether/simfactory/etc/defs.ini Cactus_Gauss/simfactory/etc/defs.ini --- Cactus_Noether/simfactory/etc/defs.ini 2013-12-08 11:53:15.000000000 +0900 +++ Cactus_Gauss/simfactory/etc/defs.ini 2013-12-08 11:40:25.000000000 +0900 @@ -16,6 +16,7 @@ simfactory src utils +thornlists EOT
# Official Cactus, SimFactory, and GetComponents entries Only in Cactus_Noether/simfactory/etc: defs.local.ini.noether Only in Cactus_Gauss/simfactory/etc: defs.local.ini~ Binary files Cactus_Noether/simfactory/lib/archive/__init__.pyc and Cactus_Gauss/simfactory/lib/archive/__init__.pyc differ Binary files Cactus_Noether/simfactory/lib/archive/petashare.pyc and Cactus_Gauss/simfactory/lib/archive/petashare.pyc differ Binary files Cactus_Noether/simfactory/lib/archive/uberftp.pyc and Cactus_Gauss/simfactory/lib/archive/uberftp.pyc differ Binary files Cactus_Noether/simfactory/lib/libutil.pyc and Cactus_Gauss/simfactory/lib/libutil.pyc differ Binary files Cactus_Noether/simfactory/lib/pyini.pyc and Cactus_Gauss/simfactory/lib/pyini.pyc differ Binary files Cactus_Noether/simfactory/lib/restartlib.pyc and Cactus_Gauss/simfactory/lib/restartlib.pyc differ Binary files Cactus_Noether/simfactory/lib/sim-build.pyc and Cactus_Gauss/simfactory/lib/sim-build.pyc differ Binary files Cactus_Noether/simfactory/lib/sim-info.pyc and Cactus_Gauss/simfactory/lib/sim-info.pyc differ Binary files Cactus_Noether/simfactory/lib/sim-manage.pyc and Cactus_Gauss/simfactory/lib/sim-manage.pyc differ Binary files Cactus_Noether/simfactory/lib/sim-sync.pyc and Cactus_Gauss/simfactory/lib/sim-sync.pyc differ Binary files Cactus_Noether/simfactory/lib/sim-util.pyc and Cactus_Gauss/simfactory/lib/sim-util.pyc differ Binary files Cactus_Noether/simfactory/lib/simarchive.pyc and Cactus_Gauss/simfactory/lib/simarchive.pyc differ Binary files Cactus_Noether/simfactory/lib/simdb.pyc and Cactus_Gauss/simfactory/lib/simdb.pyc differ Binary files Cactus_Noether/simfactory/lib/simdt.pyc and Cactus_Gauss/simfactory/lib/simdt.pyc differ Binary files Cactus_Noether/simfactory/lib/simenv.pyc and Cactus_Gauss/simfactory/lib/simenv.pyc differ Binary files Cactus_Noether/simfactory/lib/simlib.pyc and Cactus_Gauss/simfactory/lib/simlib.pyc differ Binary files Cactus_Noether/simfactory/lib/simopts.pyc and Cactus_Gauss/simfactory/lib/simopts.pyc differ Binary files Cactus_Noether/simfactory/lib/simproperties.pyc and Cactus_Gauss/simfactory/lib/simproperties.pyc differ Binary files Cactus_Noether/simfactory/lib/simremote.pyc and Cactus_Gauss/simfactory/lib/simremote.pyc differ Binary files Cactus_Noether/simfactory/lib/simrestart.pyc and Cactus_Gauss/simfactory/lib/simrestart.pyc differ Binary files Cactus_Noether/simfactory/lib/simsubs.pyc and Cactus_Gauss/simfactory/lib/simsubs.pyc differ diff -ur Cactus_Noether/simfactory/mdb/machines/datura.ini Cactus_Gauss/simfactory/mdb/machines/datura.ini --- Cactus_Noether/simfactory/mdb/machines/datura.ini 2013-12-08 11:53:11.000000000 +0900 +++ Cactus_Gauss/simfactory/mdb/machines/datura.ini 2013-12-08 11:40:25.000000000 +0900 @@ -15,7 +15,7 @@ hostname = login-damiana.aei.mpg.de rsynccmd = /home/eschnett/rsync-3.0.9/bin/rsync aliaspattern = ^(((login-)?damiana)|(sl-\d\d))(.aei.mpg.de|.damiana.admin)?$ -envsetup = export INTEL_LICENSE_FILE=/cluster/intel/licenses; export LM_LICENSE_FILE=28518@vlicense.aei.mpg.de +envsetup = source /etc/profile && export INTEL_LICENSE_FILE=/cluster/intel/licenses; export LM_LICENSE_FILE=28518@vlicense.aei.mpg.de
# Source tree management sourcebasedir = /home/@USER@/datura diff -ur Cactus_Noether/simfactory/mdb/machines/mike.ini Cactus_Gauss/simfactory/mdb/machines/mike.ini --- Cactus_Noether/simfactory/mdb/machines/mike.ini 2013-12-08 11:53:12.000000000 +0900 +++ Cactus_Gauss/simfactory/mdb/machines/mike.ini 2013-12-08 11:40:25.000000000 +0900 @@ -14,7 +14,7 @@ # Access to this machine hostname = mike1.hpc.lsu.edu rsynccmd = /usr/bin/rsync -aliaspattern = ^mike1(.hpc.lsu.edu)?$ +aliaspattern = ^mike[0-9]+(.hpc.lsu.edu)?$
# Source tree management sourcebasedir = /project/@USER@ diff -ur Cactus_Noether/simfactory/mdb/machines/stampede.ini Cactus_Gauss/simfactory/mdb/machines/stampede.ini --- Cactus_Noether/simfactory/mdb/machines/stampede.ini 2014-01-23 22:06:18.000000000 +0900 +++ Cactus_Gauss/simfactory/mdb/machines/stampede.ini 2014-01-18 23:01:24.000000000 +0900 @@ -17,7 +17,7 @@ # Access to this machine hostname = stampede.tacc.utexas.edu rsynccmd = /home1/00507/eschnett/rsync-3.0.9/bin/rsync -envsetup = module load intel/13.1.1.163 && module unload mvapich2 && module load impi/4.1.1.036 && module load papi +envsetup = module load intel/13.1.1.163 && module load papi aliaspattern = ^login[1234](.stampede.tacc.utexas.edu)?$
# Source tree management @@ -50,9 +50,9 @@ # McLachlan/ML_BSSN_CL_Helper # McLachlan/ML_WaveToy_CL EOT -optionlist = stampede.cfg +optionlist = stampede-mvapich2.cfg submitscript = stampede.sub -runscript = stampede.run +runscript = stampede-mvapich2.run make = make -j8
# Simulation management @@ -101,3 +101,17 @@ #stdout = cat @SIMULATION_NAME@.out #stderr = cat @SIMULATION_NAME@.err #stdout-follow = tail -n 100 -f @SIMULATION_NAME@.out @SIMULATION_NAME@.err + +# Intel MPI: +# +# Create a configuration using +# +# --optionlist stampede-impi.cfg --runscript stampede-impi.run +# +# if you want to use Intel MPI, or set +# +# optionlist = stampede-impi.cfg +# runscript = stampede-impi.run +# +# in your defs.local.ini for this to be the default for all configurations. + Only in Cactus_Gauss/simfactory/mdb/optionlists: stampede-impi.cfg Only in Cactus_Gauss/simfactory/mdb/optionlists: stampede-mvapich2.cfg Only in Cactus_Noether/simfactory/mdb/optionlists: stampede.cfg Only in Cactus_Gauss/simfactory/mdb/runscripts: stampede-impi.run Only in Cactus_Gauss/simfactory/mdb/runscripts: stampede-mvapich2.run Only in Cactus_Noether/simfactory/mdb/runscripts: stampede.run
***********
I could not reproduce my original claim that the Noether version with the Gauss stampede.ini worked for emails. So please forget it.
But indeed it is strange that Gauss works and Noether does not, with just such differences as above! Can you reproduce this?
Luca
On 1/23/14 7:58 PM, Ian Hinder wrote:
On 23 Jan 2014, at 02:51, Erik Schnetter schnetter@cct.lsu.edu wrote:
Luca
The emails depend on the settings in the submit script, i.e. the file simfactory/mdb/submitscripts/stampede.sub. The file "stampede.ini" should not matter.
I just checked, and there is nothing different between the stampede.ini files for the two versions that should affect whether the job emails are sent. There is also no change at all between the submit scripts between the two versions.
Assuming that you have two simulations, one where the emails are sent, and one where they are not, can you perform a diff between the recorded submission script in the simulation SIMFACTORY directory between the two simulations and post the output to the list?
If you are using the versions from the repository, you should get:
--- a/mdb/machines/stampede.ini +++ b/mdb/machines/stampede.ini @@ -1,6 +1,6 @@ [stampede]
-# last-tested-on: 2013-04-29 +# last-tested-on: 2013-11-04 # last-tested-by: Erik Schnetter schnetter@gmail.com
# NOTE: This machine configuration uses only the regular CPUs of @@ -17,7 +17,7 @@ status = experimental # Access to this machine hostname = stampede.tacc.utexas.edu rsynccmd = /home1/00507/eschnett/rsync-3.0.9/bin/rsync -envsetup = module unload mvapich2 && module load impi +envsetup = module load intel/13.1.1.163 && module unload mvapich2 && module load impi/4.1.1.036 && module load papi aliaspattern = ^login[1234](.stampede.tacc.utexas.edu)?$
# Source tree management @@ -40,6 +40,16 @@ disabled-thorns = <<EOT LSUDevelopment/WaveToyNoGhostsPETSc TAT/TATPETSc EOT +enabled-thorns = <<EOT +# CactusTest/TestAllTypes +# ExternalLibraries/OpenCL +# CactusExamples/WaveToyOpenCL +# CactusUtils/Accelerator +# CactusUtils/OpenCLRunTime +# McLachlan/ML_BSSN_CL +# McLachlan/ML_BSSN_CL_Helper +# McLachlan/ML_WaveToy_CL +EOT optionlist = stampede.cfg submitscript = stampede.sub runscript = stampede.run @@ -76,15 +86,15 @@ nodes = 6400 min-ppn = 16 allocation = NO_ALLOCATION queue = normal # [normal, large, development] -maxwalltime = 24:00:00 # development has 4:0:0 +maxwalltime = 48:00:00 # development has 4:0:0 maxqueueslots = 49 -submit = sbatch @SCRIPTFILE@ +submit = sbatch @SCRIPTFILE@; sleep 60 getstatus = squeue -j @JOB_ID@ stop = scancel @JOB_ID@ submitpattern = Submitted batch job ([0-9]+) -statuspattern = ' @JOB_ID@ ' +statuspattern = '@JOB_ID@ ' queuedpattern = ' PD ' -runningpattern = ' R ' +runningpattern = ' (CF|CG|R|TO) ' holdingpattern = ' S ' #exechost = head -n 1 SIMFACTORY/NODES #exechostpattern = ^(\S+)
Since you say that changing the stampede.ini file causes the emails to appear, please can you also post the diff between the two stampede.ini files that you are using?
Maybe there is some weird interaction between the queuing system and some environment settings in stampede.ini, e.g. an environment variable set by the modules.
I never receive email notifications from stampede, even though I am using the default submission script. I assumed it was just broken.
-erik
On Jan 22, 2014, at 20:41 , Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:
On 1/20/14 10:26 PM, Ian Hinder wrote:
On 20 Jan 2014, at 14:23, Yosef Zlochower <yosef@astro.rit.edu mailto:yosef@astro.rit.edu> wrote:
On 01/20/2014 08:06 AM, Ian Hinder wrote:
On 20 Jan 2014, at 06:14, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:
> Hello all, > > On Thursday morning, I pulled a fresh checkout of the > newest version of the Einstein Toolkit (ET_2013_11) to > use with RIT's LazEv code. I compiled it on stampede > using the current stampede.cfg located in > simfactory/mdb/optionlists which uses Intel MPI version > 4.1.0.030 and the intel compilers version 13.1.1.163 > (enabled through a module load). I submitted a short job > which I ran previously with ET_2013_05. The results come > out the same. However, the run speed as reported in > Carpet::physical_time_per_hour is poor. It starts off > good, approximately the same as with the previous build, > but over time drops to as low as half the speed over 24 > hours of evolution. On recovery from checkpoint, the > speed is even worse, dropping to below 1/4 of the > original run speed. > > So, I tried using the previous stampede.cfg included in > the ET_2013_05 branch of simfactory, the same one I used > to compile my ET_2013_05 build. This cfgfile uses the > same version of IMPI but different Intel compilers > (version 13.0.2.146). The run speed shows the same trends > as when using the newer config file. Hi Jim,
I'm quite confused by this problem report. I guess that you are meaning the following:
- You get the slowdown with the current ET_2013_11 release
- You don't get the slowdown with the ET_2013_05 release -
You do get the slowdown if you use the current ET_2013_11 release with the ET_2013_05 stampede.cfg
Is that correct?
I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).
Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node do your production runs typically use?
I'm using exactly the default simfactory config from the current trunk, so you can see the modules etc there. Checkpointing (and recovery works fine). I usually aim for something like 75% memory usage for production runs.
Hello, I would like to report a different problem with the simfactory settings for stampede: with ET Noether or trunk the job start/end emails are not sent (or at least they do not reach the Osaka University server; I had the systems administrators check). I receive the emails if I use the simfactory of Gauss. In particular, if I copy just the stampede.ini from Gauss to Noether (and no other files) and recompile, I do receive the emails.
Luca
_______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu/.
_______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On Thu, Jan 23, 2014 at 10:11:40PM +0900, Luca Baiotti wrote:
OK, so the emails not being sent is a known problem.
Here is the diff between my Gauss simfactory (emails working) and my Noether simfactory (emails not working). Both are repository versions.
Can you look at the difference of the post-processed SubmitScript files of one simulation that did send emails, and one that didn't? You should be able to find them in simulations/NAME/output-0000/SIMFACTORY/ .
Frank
On 1/23/14 11:52 PM, Frank Loeffler wrote:
On Thu, Jan 23, 2014 at 10:11:40PM +0900, Luca Baiotti wrote:
OK, so the emails not being sent is a known problem.
Here is the diff between my Gauss simfactory (emails working) and my Noether simfactory (emails not working). Both are repository versions.
Can you look at the difference of the post-processed SubmitScript files of one simulation that did send emails, and one that didn't? You should be able to find them in simulations/NAME/output-0000/SIMFACTORY/ .
It is below. Indeed non --mail entry is present in the Noether file.
Thank you, Luca
diff ETGausstest/output-0000/SIMFACTORY/SubmitScript
ETNoethertest/output-0000/SIMFACTORY/SubmitScript 7c7 < #SBATCH -J ETGausstest ---
#SBATCH -J ETNoetherte
9,13c9,12 < #SBATCH --mail-user=baiotti@ile.osaka-u.ac.jp < #SBATCH -o /scratch/01154/baiotti/simulations/ETGausstest/output-0000/ETGausstest.out < #SBATCH -e /scratch/01154/baiotti/simulations/ETGausstest/output-0000/ETGausstest.err < cd /work/01154/baiotti/Cactus_Gauss < /work/01154/baiotti/Cactus_Gauss/simfactory/bin/sim run ETGausstest --machine=stampede --restart-id=0 ---
#SBATCH -o
/scratch/01154/baiotti/simulations/ETNoethertest/output-0000/ETNoethertest.out
#SBATCH -e
/scratch/01154/baiotti/simulations/ETNoethertest/output-0000/ETNoethertest.err
cd /work/01154/baiotti/Cactus_Noether /work/01154/baiotti/Cactus_Noether/simfactory/bin/sim run
ETNoethertest --machine=stampede --restart-id=0
On 24 Jan 2014, at 01:15, Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:
On 1/23/14 11:52 PM, Frank Loeffler wrote:
On Thu, Jan 23, 2014 at 10:11:40PM +0900, Luca Baiotti wrote:
OK, so the emails not being sent is a known problem.
Here is the diff between my Gauss simfactory (emails working) and my Noether simfactory (emails not working). Both are repository versions.
Can you look at the difference of the post-processed SubmitScript files of one simulation that did send emails, and one that didn't? You should be able to find them in simulations/NAME/output-0000/SIMFACTORY/ .
It is below. Indeed non --mail entry is present in the Noether file.
The Gauss version looks like this:
#! /bin/bash #SBATCH -A @ALLOCATION@ #SBATCH -p @QUEUE@ #SBATCH -t @WALLTIME@ #SBATCH -N @NODES@ -n @NUM_PROCS@ #SBATCH @("@CHAINED_JOB_ID@" != "" ? "-d afterany:@CHAINED_JOB_ID@" : "")@ #SBATCH -J @SHORT_SIMULATION_NAME@ #SBATCH --mail-type=ALL #SBATCH -o @RUNDIR@/@SIMULATION_NAME@.out #SBATCH -e @RUNDIR@/@SIMULATION_NAME@.err cd @SOURCEDIR@ @SIMFACTORY@ run @SIMULATION_NAME@ --machine=@MACHINE@ --restart-id=@RESTART_ID@ @FROM_RESTART_COMMAND@
If you have anything different, then maybe you added it yourself? Can do do an "svn diff" on the submit script in your Gauss version? I think what has happened is that you fixed the problem in your Gauss version by adding the lines
#SBATCH --mail-user=baiotti@ile.osaka-u.ac.jp
This would explain why mail notifications are not working for me: they have never worked in the committed version.
On 1/24/14 6:42 PM, Ian Hinder wrote:
On 24 Jan 2014, at 01:15, Luca Baiotti <baiotti@ile.osaka-u.ac.jp mailto:baiotti@ile.osaka-u.ac.jp> wrote:
On 1/23/14 11:52 PM, Frank Loeffler wrote:
On Thu, Jan 23, 2014 at 10:11:40PM +0900, Luca Baiotti wrote:
OK, so the emails not being sent is a known problem.
Here is the diff between my Gauss simfactory (emails working) and my Noether simfactory (emails not working). Both are repository versions.
Can you look at the difference of the post-processed SubmitScript files of one simulation that did send emails, and one that didn't? You should be able to find them in simulations/NAME/output-0000/SIMFACTORY/ .
It is below. Indeed non --mail entry is present in the Noether file.
The Gauss version looks like this:
#! /bin/bash #SBATCH -A @ALLOCATION@ #SBATCH -p @QUEUE@ #SBATCH -t @WALLTIME@ #SBATCH -N @NODES@ -n @NUM_PROCS@ #SBATCH @("@CHAINED_JOB_ID@" != "" ? "-d afterany:@CHAINED_JOB_ID@" : "")@ #SBATCH -J @SHORT_SIMULATION_NAME@ #SBATCH --mail-type=ALL #SBATCH -o @RUNDIR@/@SIMULATION_NAME@.out #SBATCH -e @RUNDIR@/@SIMULATION_NAME@.err cd @SOURCEDIR@ @SIMFACTORY@ run @SIMULATION_NAME@ --machine=@MACHINE@ --restart-id=@RESTART_ID@ @FROM_RESTART_COMMAND@
If you have anything different, then maybe you added it yourself? Can do do an "svn diff" on the submit script in your Gauss version? I think what has happened is that you fixed the problem in your Gauss version by adding the lines
#SBATCH --mail-user=baiotti@ile.osaka-u.ac.jp mailto:mail-user=baiotti@ile.osaka-u.ac.jp
This would explain why mail notifications are not working for me: they have never worked in the committed version.
You are right! I had modified it long ago and forgot. Then I assumed that simfactory worked properly on its own... And to complicate things, I had recently removed the original simfactory directory in my Cactus_Gauss and checked out a new one (so svn diff didn't show any difference).
In conclusion, can the script for stampede be fixed in the repository?
Thank you, Luca
On 24 Jan 2014, at 12:42, Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:
On 1/24/14 6:42 PM, Ian Hinder wrote:
On 24 Jan 2014, at 01:15, Luca Baiotti <baiotti@ile.osaka-u.ac.jp mailto:baiotti@ile.osaka-u.ac.jp> wrote:
On 1/23/14 11:52 PM, Frank Loeffler wrote:
On Thu, Jan 23, 2014 at 10:11:40PM +0900, Luca Baiotti wrote:
OK, so the emails not being sent is a known problem.
Here is the diff between my Gauss simfactory (emails working) and my Noether simfactory (emails not working). Both are repository versions.
Can you look at the difference of the post-processed SubmitScript files of one simulation that did send emails, and one that didn't? You should be able to find them in simulations/NAME/output-0000/SIMFACTORY/ .
It is below. Indeed non --mail entry is present in the Noether file.
The Gauss version looks like this:
#! /bin/bash #SBATCH -A @ALLOCATION@ #SBATCH -p @QUEUE@ #SBATCH -t @WALLTIME@ #SBATCH -N @NODES@ -n @NUM_PROCS@ #SBATCH @("@CHAINED_JOB_ID@" != "" ? "-d afterany:@CHAINED_JOB_ID@" : "")@ #SBATCH -J @SHORT_SIMULATION_NAME@ #SBATCH --mail-type=ALL #SBATCH -o @RUNDIR@/@SIMULATION_NAME@.out #SBATCH -e @RUNDIR@/@SIMULATION_NAME@.err cd @SOURCEDIR@ @SIMFACTORY@ run @SIMULATION_NAME@ --machine=@MACHINE@ --restart-id=@RESTART_ID@ @FROM_RESTART_COMMAND@
If you have anything different, then maybe you added it yourself? Can do do an "svn diff" on the submit script in your Gauss version? I think what has happened is that you fixed the problem in your Gauss version by adding the lines
#SBATCH --mail-user=baiotti@ile.osaka-u.ac.jp mailto:mail-user=baiotti@ile.osaka-u.ac.jp
This would explain why mail notifications are not working for me: they have never worked in the committed version.
You are right! I had modified it long ago and forgot. Then I assumed that simfactory worked properly on its own... And to complicate things, I had recently removed the original simfactory directory in my Cactus_Gauss and checked out a new one (so svn diff didn't show any difference).
In conclusion, can the script for stampede be fixed in the repository?
I have changed it in my version, so if it works the next time I submit something, I'll commit it to the repository.
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Ian Hinder -
James Healy -
Luca Baiotti -
Roland Haas -
Yosef Zlochower