<div dir="ltr">Ian<div><br></div><div>It seems then it would be a good idea update the Stampede parameter file in Simfactory to use mvapich instead of impi -- do you agree?</div><div><br></div><div>-erik</div><div><br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 3, 2013 at 12:38 PM, Ian Hinder <span dir="ltr"><<a href="mailto:ian.hinder@aei.mpg.de" target="_blank">ian.hinder@aei.mpg.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div><div class="h5"><div>On 3 Jun 2013, at 18:02, Roland Haas <<a href="mailto:roland.haas@physics.gatech.edu" target="_blank">roland.haas@physics.gatech.edu</a>> wrote:</div>
<br><blockquote type="cite">Present: Erik, Peter, Ian, Yosef, Matt, Roland<br><br>automated tests:<br><br>* want to run tests periodically on production machines rather than just<br>in Ubuntu VM to test other compilers/MPI implementation than gcc/OpenMPI<br>
* Ian has setup for Datura under development, add other machines once<br>system is stable. This method runs Jenkins on the head node.<br>* add short (20min or so) simulation to tests to check more complex setups<br><br>ET workshop:<br>
* David Rideout offered to give a presentation (private email to Erik,<br>hope I did not get this wrong)<br>* goals:<br>** ES wants to improve Carpet scalability to >10^4-10^5 processors<br>** (known) bottlenecks are: bboxset class, any lower-d IO which<br>
serializes it seems<br>* PLEASE think of topics for the workshop and post them to the mailing list<br><br>occasional hangs in MPI reported by Yosef:<br>* hangs occur with or without the parity thorn<br>* hangs occur on RIT cluster (with parity thorn, mvapich)<br>
* hangs occur on Stampede (no parity thorn, IMPI)<br>* Ian saw codes hang in ibrun when MPI versions did not match, this<br>seems to not be the case here<br></blockquote><div><br></div></div></div><div>This refers to using a slightly different version of Intel MPI when building to the one used at runtime (because simfactory didn't load the module at the time, so it was using what was in the environment; this has now been fixed). In the end, this was not the cause of the hangs. The hangs persisted after this fix. To eliminate the hangs (which happened immediately at startup), I switched from Intel MPI to mvapich on Stampede. I have also experienced hangs during the simulation with Intel MPI on Supermuc; there, switching to IBM MPI fixed it.</div>
<span class="HOEnZb"><font color="#888888"><div><br></div></font></span></div><span class="HOEnZb"><font color="#888888"><div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder" target="_blank">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></font></span></div><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>