On 2013-09-21, at 12:13 , Ian Hinder ian.hinder@aei.mpg.de wrote:
Hi,
Does anyone have a suggestion for how I can make a run recover if HDF5 claims to be out of memory? The original run was on 1056 cores of stampede. Recovery fails both on the same number of cores as well as on 2048. Would a newer version of HDF5 help? Is there some parameter I can set? I am already using CarpetIOHDF5::open_one_input_file_at_a_time = yes. I am using CarpetIOHDF5::compression_level = 9. I wonder if the problem is made worse by having to decompress when reading the file. Logically, I don't see any reason why HDF5 should need more memory to recover than the original run used.
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0: #000: H5Dio.c line 174 in H5Dread(): can't read data major: Dataset minor: Read failed #001: H5Dio.c line 449 in H5D__read(): can't read data major: Dataset minor: Read failed #002: H5Dchunk.c line 1735 in H5D__chunk_read(): unable to read raw data chunk major: Low-level I/O minor: Read failed #003: H5Dchunk.c line 2766 in H5D__chunk_lock(): data pipeline read failed major: Data filters minor: Filter operation failed #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure during read major: Data filters minor: Read failed #005: H5Zdeflate.c line 136 in H5Z_filter_deflate(): memory allocation failed for deflate uncompression major: Resource unavailable minor: No space available for allocation WARNING[L1,P254] (CarpetIOHDF5): : HDF5 call 'H5Dread (dataset, datatype, memspace, filespace, xfer, cctkGH->data[patch->vindex][timelevel])' returned error code -1
What is the size of this chunk? Each chunk needs to be decompressed at once; using a smaller chunk size may help. You may be able to modify the chunk size via h5repack (or disable decompression).
I think there are also options or properties one can set to determine how much memory HDF5 uses for its caches. I don't know much about these; an HDF5 mailing list or help ticket may tell you more.
-erik