[ET Trac] [Einstein Toolkit] #1878: hdf5 deflate fragments memory

[Einstein Toolkit]

#1878: hdf5 deflate fragments memory
-------------------------+--------------------------------------------------
 Reporter:  knarf        |       Owner:                     
     Type:  enhancement  |      Status:  new                
 Priority:  minor        |   Milestone:                     
Component:  Other        |     Version:  development version
 Keywords:  HDF5         |  
-------------------------+--------------------------------------------------
 In ticket #1874, a problem with memory management of the HDF5 zlib deflate
 filter was mentioned. This ticket here was opened to collect information
 about this problem, with the aim of eventually providing a patch to HDF5,
 submitting a ticket to the HDF5 developers, and possibly implementing that
 change as patch just for the ET.

 The problem is that HDF5 allocates while deflating an output buffer (for
 the uncompressed data) of the size of the compressed data, and then makes
 it larger (by a factor of two - another problem), if needed (almost
 always).

 This happens in H5Z_filter_deflate() in H5Zdeflate.c . One way to solve
 this would be to use the z_strm.total_in value, which holds the size of
 the uncompressed data, saved by the compressor. However, I am currently
 unsure whether this is a value we can always rely on. The format
 specification mentions it as required field, while the zlib documentation
 talks about it as "may have been saved by the compressor". I didn't look
 at any actual data.

 In any case, if this value is present, we can use it to allocate the
 output buffer of the correct size. If it isn't present, the best we could
 do is at least not increase the size by a factor of two, which is known to
 cause memory fragmentation.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1878>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit