Comment (by Roland Haas):

Since the issue is caused by code in libhdf5 there is not much we can actually do about this. Even patching H5Z_filter_deflate in the tar file included in ExternalLibraries/HDF5 won’t fix the issue on most clusters since on most clusters we use the system installed HDF5 library. Looking at the API call https://support.hdfgroup.org/documentation/hdf5/latest/group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d there is no option to specify the initial buffer size.

A patch for the self-build library is in rhaas/deflate of https://github.com/einsteinToolkit/ExternalLibraries-hdf5 for all the good it might do. The 32bit size field in gzip files is no longer considered authorative (fails to record uncompressed file sizes >2GB).

I’d suggest to close this as “wontfix” since there is nothing that we can really do about this.

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/1878/hdf5-deflate-fragments-memory