Hello all,
It's actually worse than that. TACC's petsc module is linked against their phdf5 library (but the module does not require it so module load petsc does not automatically pull it in or complain if a conflicting version is loaded), ie:
rhaas@login4:~/work/ET_trunk/simfactory$ module list Currently Loaded Modules: 1) TACC-paths 3) cluster-paths 5) cluster 7) intel/13.1.1.163 9) petsc/3.5 2) Linux 4) xalt/0.4.6 6) TACC 8) mvapich2/1.9a2 10) papi/5.3.0 (m)
rhaas@login4:~/work/ET_trunk/simfactory$ ldd $TACC_PETSC_LIB/libpetsc.so.3.5 | grep hdf5 libhdf5_fortran.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5_fortran.so.7 (0x00002ae35eb38000) libhdf5_hl.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5_hl.so.7 (0x00002ae35ed7c000) libhdf5.so.7 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libhdf5.so.7 (0x00002ae35efaf000) libsz.so.2 => /opt/apps/intel13/mvapich2_1_9/phdf5/1.8.9/lib/libsz.so.2 (0x00002ae364089000)
so of
(a) disable petsc (b) use the phdf5 module (this can be annoying since then every utility that uses hdf5 uses ibrun and creates a unique ibrun/slurm control file in $HOME/.slurm that is *not* ever deleted and can lead to quota violations due to number of files in $HOME) (c) check our own compiled HDF5 to make sure it builds the parallel version if requested (no option to request this exists yet) (d) (if this is possible) build our own petsc without parallel hdf5 support
(c) is no longer viable since there'd be a comflict between the version petsc was compiled with and our hdf5 version (even if we can get the dynamic linker to use "our" version which we don't want to since we don not want to build dynamic libraries).
Yours, Roland