<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 20 Jan 2014, at 14:23, Yosef Zlochower <<a href="mailto:yosef@astro.rit.edu">yosef@astro.rit.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On 01/20/2014 08:06 AM, Ian Hinder wrote:<br><blockquote type="cite">On 20 Jan 2014, at 06:14, James Healy <<a href="mailto:jchsma@rit.edu">jchsma@rit.edu</a>> wrote:<br><br><blockquote type="cite">Hello all,<br><br>On Thursday morning, I pulled a fresh checkout of the newest version of<br>the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I<br>compiled it on stampede using the current stampede.cfg located in<br>simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and<br>the intel compilers version 13.1.1.163 (enabled through a module load).<br>I submitted a short job which I ran previously with ET_2013_05. The<br>results come out the same. However, the run speed as reported in<br>Carpet::physical_time_per_hour is poor. It starts off good,<br>approximately the same as with the previous build, but over time drops<br>to as low as half the speed over 24 hours of evolution. On recovery from<br>checkpoint, the speed is even worse, dropping to below 1/4 of the<br>original run speed.<br><br>So, I tried using the previous stampede.cfg included in the ET_2013_05<br>branch of simfactory, the same one I used to compile my ET_2013_05<br>build. This cfgfile uses the same version of IMPI but different Intel<br>compilers (version 13.0.2.146). The run speed shows the same trends as<br>when using the newer config file.<br></blockquote>Hi Jim,<br><br>I'm quite confused by this problem report. I guess that you are meaning the following:<br><br>- You get the slowdown with the current ET_2013_11 release<br>- You don't get the slowdown with the ET_2013_05 release<br>- You do get the slowdown if you use the current ET_2013_11 release with the ET_2013_05 stampede.cfg<br><br>Is that correct?<br><br>I consider Intel MPI to be unusable on Stampede, and that it always has been. I used to get random crashes, hangs and slowdowns. I also experienced similar problems with Intel MPI on SuperMUC. For any serious work, I have always used MVAPICH2 on Stampede. In the current ET trunk Intel MPI has been replaced with MVAPICH2. I would try the current trunk and see if this fixes your problems. You can also use just the stampede files from the current trunk with the ET_2013_11 release (make sure you use the ones listed in stampede.ini).<br></blockquote>Interesting. I haven't been able to get a run to work with mvapich2 because of an issue with the runs<br>dying during checkpoint. Which config file are you using (module loaded, etc)? How much ram per node<br>do your production runs typically use?<br></blockquote><div><br></div><div>I'm using exactly the default simfactory config from the current trunk, so you can see the modules etc there. Checkpointing (and recovery works fine). I usually aim for something like 75% memory usage for production runs.</div><div><br></div><br><blockquote type="cite"><br><blockquote type="cite">We didn't change the MPI version before the release, as that would have been quite an invasive change at that point. However, I would consider backporting this, after suitable discussion.<br><br>Of course, your problem might be unrelated to the version of MPI. I am running perfectly fine on stampede with the current trunk (MVAPICH2); runs have a consistent speed and retain this speed after recovery.<br><br></blockquote><br></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></body></html>