#1878: hdf5 deflate fragments memory -------------------------+-------------------------------------------------- Reporter: knarf | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Other | Version: development version Keywords: HDF5 | -------------------------+-------------------------------------------------- In ticket #1874, a problem with memory management of the HDF5 zlib deflate filter was mentioned. This ticket here was opened to collect information about this problem, with the aim of eventually providing a patch to HDF5, submitting a ticket to the HDF5 developers, and possibly implementing that change as patch just for the ET.
The problem is that HDF5 allocates while deflating an output buffer (for the uncompressed data) of the size of the compressed data, and then makes it larger (by a factor of two - another problem), if needed (almost always).
This happens in H5Z_filter_deflate() in H5Zdeflate.c . One way to solve this would be to use the z_strm.total_in value, which holds the size of the uncompressed data, saved by the compressor. However, I am currently unsure whether this is a value we can always rely on. The format specification mentions it as required field, while the zlib documentation talks about it as "may have been saved by the compressor". I didn't look at any actual data.
In any case, if this value is present, we can use it to allocate the output buffer of the correct size. If it isn't present, the best we could do is at least not increase the size by a factor of two, which is known to cause memory fragmentation.