Hello all,
- GRHydro failing tests on stampede: Roland to check what is going on
and to fix
It seems that this is not really a GRHydro issue. The code crashes in HDF5's init routine (called once whenever any other HDF5 routine is used). GRHydro may just trigger this because it relies on EOS_Omni which uses HDF5.
I have not yet fully cornered the culprit but there are some iffy things in the current stampede configuration files. We load the petsc module and petsc normally requires parallel HDF5 (which rules out the C++ interface) but I am not sure if our own built hdf5 library enables this or uses the correct MPI compiler. Also envsetup is rather sparse and loads the intel 13 compiler and petsc but does not unload any other potentially conflicting packages.
Switching the BUILD to /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9 and loading the phdf5 module produces and executable that can run the tests. It right now fails in the hdf5 utilities since they do not find MPI anymore (which is odd since HDF5 should have and OPTIONAL MPI in configure.ccl for exactly this reason).
So possible solutions may be:
(a) disable petsc (b) use the phdf5 module (this can be annoying since then every utility that uses hdf5 uses ibrun and creates a unique ibrun/slurm control file in $HOME/.slurm that is *not* ever deleted and can lead to quota violations due to number of files in $HOME) (c) check our own compiled HDF5 to make sure it builds the parallel version if requested (no option to request this exists yet) (d) (if this is possible) build our own petsc without parallel hdf5 support
I will be offline for the next 6 hours, so if someone else wants to give it a try, feel free to do so.
Yours, Roland