Present: Erik, Peter, Ian, Yosef, Matt, Roland
automated tests:
* want to run tests periodically on production machines rather than just in Ubuntu VM to test other compilers/MPI implementation than gcc/OpenMPI * Ian has setup for Datura under development, add other machines once system is stable. This method runs Jenkins on the head node. * add short (20min or so) simulation to tests to check more complex setups
ET workshop: * David Rideout offered to give a presentation (private email to Erik, hope I did not get this wrong) * goals: ** ES wants to improve Carpet scalability to >10^4-10^5 processors ** (known) bottlenecks are: bboxset class, any lower-d IO which serializes it seems * PLEASE think of topics for the workshop and post them to the mailing list
occasional hangs in MPI reported by Yosef: * hangs occur with or without the parity thorn * hangs occur on RIT cluster (with parity thorn, mvapich) * hangs occur on Stampede (no parity thorn, IMPI) * Ian saw codes hang in ibrun when MPI versions did not match, this seems to not be the case here
Yours, Roland
On 3 Jun 2013, at 18:02, Roland Haas roland.haas@physics.gatech.edu wrote:
Present: Erik, Peter, Ian, Yosef, Matt, Roland
automated tests:
- want to run tests periodically on production machines rather than just
in Ubuntu VM to test other compilers/MPI implementation than gcc/OpenMPI
- Ian has setup for Datura under development, add other machines once
system is stable. This method runs Jenkins on the head node.
- add short (20min or so) simulation to tests to check more complex setups
ET workshop:
- David Rideout offered to give a presentation (private email to Erik,
hope I did not get this wrong)
- goals:
** ES wants to improve Carpet scalability to >10^4-10^5 processors ** (known) bottlenecks are: bboxset class, any lower-d IO which serializes it seems
- PLEASE think of topics for the workshop and post them to the mailing list
occasional hangs in MPI reported by Yosef:
- hangs occur with or without the parity thorn
- hangs occur on RIT cluster (with parity thorn, mvapich)
- hangs occur on Stampede (no parity thorn, IMPI)
- Ian saw codes hang in ibrun when MPI versions did not match, this
seems to not be the case here
This refers to using a slightly different version of Intel MPI when building to the one used at runtime (because simfactory didn't load the module at the time, so it was using what was in the environment; this has now been fixed). In the end, this was not the cause of the hangs. The hangs persisted after this fix. To eliminate the hangs (which happened immediately at startup), I switched from Intel MPI to mvapich on Stampede. I have also experienced hangs during the simulation with Intel MPI on Supermuc; there, switching to IBM MPI fixed it.
Ian
It seems then it would be a good idea update the Stampede parameter file in Simfactory to use mvapich instead of impi -- do you agree?
-erik
On Mon, Jun 3, 2013 at 12:38 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 3 Jun 2013, at 18:02, Roland Haas roland.haas@physics.gatech.edu wrote:
Present: Erik, Peter, Ian, Yosef, Matt, Roland
automated tests:
- want to run tests periodically on production machines rather than just
in Ubuntu VM to test other compilers/MPI implementation than gcc/OpenMPI
- Ian has setup for Datura under development, add other machines once
system is stable. This method runs Jenkins on the head node.
- add short (20min or so) simulation to tests to check more complex setups
ET workshop:
- David Rideout offered to give a presentation (private email to Erik,
hope I did not get this wrong)
- goals:
** ES wants to improve Carpet scalability to >10^4-10^5 processors ** (known) bottlenecks are: bboxset class, any lower-d IO which serializes it seems
- PLEASE think of topics for the workshop and post them to the mailing list
occasional hangs in MPI reported by Yosef:
- hangs occur with or without the parity thorn
- hangs occur on RIT cluster (with parity thorn, mvapich)
- hangs occur on Stampede (no parity thorn, IMPI)
- Ian saw codes hang in ibrun when MPI versions did not match, this
seems to not be the case here
This refers to using a slightly different version of Intel MPI when building to the one used at runtime (because simfactory didn't load the module at the time, so it was using what was in the environment; this has now been fixed). In the end, this was not the cause of the hangs. The hangs persisted after this fix. To eliminate the hangs (which happened immediately at startup), I switched from Intel MPI to mvapich on Stampede. I have also experienced hangs during the simulation with Intel MPI on Supermuc; there, switching to IBM MPI fixed it.
-- Ian Hinder http://numrel.aei.mpg.de/people/hinder
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 4 Jun 2013, at 02:28, Erik Schnetter schnetter@cct.lsu.edu wrote:
Ian
It seems then it would be a good idea update the Stampede parameter file in Simfactory to use mvapich instead of impi -- do you agree?
From my own experiences, yes, I agree. Does anyone have any objections to this?
Hello all,
From my own experiences, yes, I agree. Does anyone have any objections to this?
No objections.
Yours, Roland
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Ian Hinder -
Roland Haas