#1874: Activate HDF5 shuffle filter ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: enhancement | Status: review Priority: unset | Milestone: Component: EinsteinToolkit thorn | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by hinder):
The HDF5 gzip decompression filter allocates a block of memory the same size as the compressed data for the uncompressed data. It then reallocs it "in case" it needs to. It will always need to. This is ridiculous, and causes excessive memory fragmentation, to the extent that recovering from a compressed checkpoint file is very difficult unless you have a large amount of extra memory, or swap available (which seems to be rarer these days on HPC systems).