#1878: hdf5 deflate fragments memory
Reporter: Frank Löffler Status: new Milestone: Version: development version Type: enhancement Priority: minor Component: Other
Comment (by Roland Haas):
Since the issue is caused by code in libhdf5 there is not much we can actually do about this. Even patching `H5Z_filter_deflate` in the tar file included in ExternalLibraries/HDF5 won’t fix the issue on most clusters since on most clusters we use the system installed HDF5 library. Looking at the API call [https://support.hdfgroup.org/documentation/hdf5/latest/group%5C_%5C_%5C_d%5C...) there is no option to specify the initial buffer size.
A patch for the self-build library is in `rhaas/deflate` of [https://github.com/einsteinToolkit/ExternalLibraries-hdf5%5D(https://github....) for all the good it might do. The 32bit size field in gzip files is no longer considered authorative (fails to record uncompressed file sizes >2GB).
I’d suggest to close this as “wontfix” since there is nothing that we can really do about this.
-- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/1878/hdf5-deflate-fragm...