#1878: hdf5 deflate fragments memory --------------------------+------------------------------------------------- Reporter: knarf | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Other | Version: development version Resolution: | Keywords: HDF5 --------------------------+-------------------------------------------------
Comment (by eschnett):
I wonder whether memory fragmentation is not a problem caused by the malloc implementation. To test this, you can test with different (newer) versions of libc, or with a malloc replacement. I recommend in particular jemalloc http://www.canonware.com/jemalloc/ -- you build the code as usual, and add {{{-ljemalloc}}} when linking. This replaces malloc, free, etc., and the equivalent C++ mechanism. jemalloc claims to be better than GNU malloc is many ways.
Another way to avoid fragmentation (that we use in CarpetIOASCII) is to allocate memory not directly via malloc, but rather from a pre-allocated pool. This pool is so large that fragmentation is not an issue on 64-bit systems, since it can be returned to the OS as a whole. Implementing a pool is not too difficult; see Carpet's mempool class (CarpetLib, mem.hh and mem.cc).