[ET Trac] #1878: hdf5 deflate fragments memory

[Roland Haas]

#1878: hdf5 deflate fragments memory

 Reporter: Frank Löffler
   Status: new
Milestone: 
  Version: development version
     Type: enhancement
 Priority: minor
Component: Other

Comment (by Roland Haas):

Since the issue is caused by code in libhdf5 there is not much we can actually do about this. Even patching `H5Z_filter_deflate` in the tar file included in ExternalLibraries/HDF5 won’t fix the issue on most clusters since on most clusters we use the system installed HDF5 library. Looking at the API call [https://support.hdfgroup.org/documentation/hdf5/latest/group\_\_\_d\_c\_p\_l.html#gaf1f569bfc54552bdb9317d2b63318a0d](https://support.hdfgroup.org/documentation/hdf5/latest/group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d) there is no option to specify the initial buffer size.  
  
A patch for the self-build library is in `rhaas/deflate` of [https://github.com/einsteinToolkit/ExternalLibraries-hdf5](https://github.com/einsteinToolkit/ExternalLibraries-hdf5) for all the good it might do. The 32bit size field in gzip files is no longer considered authorative \(fails to record uncompressed file sizes >2GB\).

I’d suggest to close this as “wontfix” since there is nothing that we can really do about this.

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/1878/hdf5-deflate-fragments-memory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/trac/attachments/20250103/f16a0832/attachment.htm>