Present: Frank, Peter, Steve, Matt, Roland, Ian, Elo, Erik, Jonah, Josh, Zach, Philipp
Tickets: * intel compiler miscompiles NewRad ** intel 15 fixes this again ** Steve will try and write an autoconf test for this * use after free in IOUtil's parameter recovery routine ** Frank to fix
* IllinoisGRMHD: ** suggest to use custom thornlist based on release thornlist but using different branches for HydroBase and Carpet
Upcoming release: ** Send out announcement of upcoming release timeline to mailing list ** Elo has new simfactory files for new machine, will test and we will include machine in release announcement if tests pass * Frank to make people do what they promise to do * GRHydro failing tests on stampede: Roland to check what is going on and to fix
Yours, Roland
Hello all,
- GRHydro failing tests on stampede: Roland to check what is going on
and to fix
It seems that this is not really a GRHydro issue. The code crashes in HDF5's init routine (called once whenever any other HDF5 routine is used). GRHydro may just trigger this because it relies on EOS_Omni which uses HDF5.
I have not yet fully cornered the culprit but there are some iffy things in the current stampede configuration files. We load the petsc module and petsc normally requires parallel HDF5 (which rules out the C++ interface) but I am not sure if our own built hdf5 library enables this or uses the correct MPI compiler. Also envsetup is rather sparse and loads the intel 13 compiler and petsc but does not unload any other potentially conflicting packages.
Switching the BUILD to /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 and loading the phdf5 module produces and executable that can run the tests. It right now fails in the hdf5 utilities since they do not find MPI anymore (which is odd since HDF5 should have and OPTIONAL MPI in configure.ccl for exactly this reason).
So possible solutions may be:
(a) disable petsc (b) use the phdf5 module (this can be annoying since then every utility that uses hdf5 uses ibrun and creates a unique ibrun/slurm control file in $HOME/.slurm that is *not* ever deleted and can lead to quota violations due to number of files in $HOME) (c) check our own compiled HDF5 to make sure it builds the parallel version if requested (no option to request this exists yet) (d) (if this is possible) build our own petsc without parallel hdf5 support
I will be offline for the next 6 hours, so if someone else wants to give it a try, feel free to do so.
Yours, Roland
Hello all,
It's actually worse than that. TACC's petsc module is linked against their phdf5 library (but the module does not require it so module load petsc does not automatically pull it in or complain if a conflicting version is loaded), ie:
rhaas@login4:~/work/ET_trunk/simfactory$ module list Currently Loaded Modules: 1) TACC-paths 3) cluster-paths 5) cluster 7) intel/13.1.1.163 9) petsc/3.5 2) Linux 4) xalt/0.4.6 6) TACC 8) mvapich2/1.9a2 10) papi/5.3.0 (m)
rhaas@login4:~/work/ET_trunk/simfactory$ ldd $TACC_PETSC_LIB/libpetsc.so.3.5 | grep hdf5 libhdf5_fortran.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5_fortran.so.7 (0x00002ae35eb38000) libhdf5_hl.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5_hl.so.7 (0x00002ae35ed7c000) libhdf5.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5.so.7 (0x00002ae35efaf000) libsz.so.2 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libsz.so.2 (0x00002ae364089000)
so of
(a) disable petsc (b) use the phdf5 module (this can be annoying since then every utility that uses hdf5 uses ibrun and creates a unique ibrun/slurm control file in $HOME/.slurm that is *not* ever deleted and can lead to quota violations due to number of files in $HOME) (c) check our own compiled HDF5 to make sure it builds the parallel version if requested (no option to request this exists yet) (d) (if this is possible) build our own petsc without parallel hdf5 support
(c) is no longer viable since there'd be a comflict between the version petsc was compiled with and our hdf5 version (even if we can get the dynamic linker to use "our" version which we don't want to since we don not want to build dynamic libraries).
Yours, Roland
If PETSc is not correctly configured on Stampede, then we should not use it. I am now trying to build it ourselves instead.
-erik
On Tue, Nov 4, 2014 at 3:35 AM, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello all,
It's actually worse than that. TACC's petsc module is linked against their phdf5 library (but the module does not require it so module load petsc does not automatically pull it in or complain if a conflicting version is loaded), ie:
rhaas@login4:~/work/ET_trunk/simfactory$ module list Currently Loaded Modules:
- TACC-paths 3) cluster-paths 5) cluster 7) intel/13.1.1.163
- petsc/3.5
- Linux 4) xalt/0.4.6 6) TACC 8) mvapich2/1.9a2
- papi/5.3.0 (m)
rhaas@login4:~/work/ET_trunk/simfactory$ ldd $TACC_PETSC_LIB/libpetsc.so.3.5 | grep hdf5 libhdf5_fortran.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5_fortran.so.7 (0x00002ae35eb38000) libhdf5_hl.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5_hl.so.7 (0x00002ae35ed7c000) libhdf5.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5.so.7 (0x00002ae35efaf000) libsz.so.2 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libsz.so.2 (0x00002ae364089000)
so of
(a) disable petsc (b) use the phdf5 module (this can be annoying since then every utility that uses hdf5 uses ibrun and creates a unique ibrun/slurm control file in $HOME/.slurm that is *not* ever deleted and can lead to quota violations due to number of files in $HOME) (c) check our own compiled HDF5 to make sure it builds the parallel version if requested (no option to request this exists yet) (d) (if this is possible) build our own petsc without parallel hdf5 support
(c) is no longer viable since there'd be a comflict between the version petsc was compiled with and our hdf5 version (even if we can get the dynamic linker to use "our" version which we don't want to since we don not want to build dynamic libraries).
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello Erik,
If PETSc is not correctly configured on Stampede, then we should not use it. I am now trying to build it ourselves instead.
Thank you. I suspect that you will have to make sure to compile with only serial hdf5, otherwise the issue with the need for ibrun for the utilities remains.
Yours, Roland
On Tue, Nov 04, 2014 at 09:31:49AM -0500, Erik Schnetter wrote:
If PETSc is not correctly configured on Stampede, then we should not use it. I am now trying to build it ourselves instead.
Did this succeed? It would be nice to be able to support PETSc on stampede from within the ET. Could user support maybe help providing a 'serial' PETSc installation?
Frank
Yes, this succeeded and has been committed.
-erik
On Thu, Nov 6, 2014 at 11:09 PM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Tue, Nov 04, 2014 at 09:31:49AM -0500, Erik Schnetter wrote:
If PETSc is not correctly configured on Stampede, then we should not use it. I am now trying to build it ourselves instead.
Did this succeed? It would be nice to be able to support PETSc on stampede from within the ET. Could user support maybe help providing a 'serial' PETSc installation?
Frank
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Roland Haas