Hi Jakob,
I had the same problem some time ago. Since your HDF5 files are
compressed, Cactus tries to unpack them before reading, and this
operation somehow is very memory-intensive. Try to repack your
checkpoint files before restarting to reduce compression level to zero.
You can use the standard h5repack utility for that. Here's a small
script to uncompress all checkpoint files and put them to an $OUTDIR
directory:
----------------
#!/bin/bash
ITER=5555 # which iteration to use
OUTDIR=$SCRATCH/tmp # output directory
for f in checkpoint.chkpt.it_${ITER}.file_*.h5; do
echo processing $f...
h5repack -i $f -o $OUTDIR/$f -f NONE
done
----------------
This will make your checkpoint files 5-10 times larger, but because they
don't need to be unpacked in RAM, less memory is required to read them.
Also, try the following:
1. set:
CarpetIOHDF5::open_one_input_file_at_a_time = yes
IO::abort_on_io_errors = yes
2. use less cores per node.
3. try running on larger number of processors.
Cheers,
- Oleg Korobkin
> ------------------------------------------------------------------------
Frank Loeffler wrote:
> Hi,
>
> On Thu, Feb 17, 2011 at 03:24:04PM +0900, Jakob Hansen wrote:
>> #005: H5Zdeflate.c line 133 in H5Z_filter_deflate(): memory allocation
>> failed for deflate uncompression
>> major: Resource unavailable
>> minor: No space available for allocation
>
>> Any ideas for possible cause and solution to this?
>
> Looking at these error messages I suggest you first do an h5ls/h5dump to
> see if the files are actually ok. If that is so, what I would suspect next
> is that reading the files takes more memory than is available, as indicated
> by the message above. In this case I suggest to try the workarounds
> mentioned in this thread:
>
> http://lists.einsteintoolkit.org/pipermail/users/2011-February/000852.html
>
> If this also doesn't help I would suggest to try again using more
> available memory, e.g. using more nodes.
>
> Frank
>
>
>
>
> _______________________________________________
> Users mailing list
> Users@einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users