Thank you for suggestions,<br><br>Oleg, your h5repack script did the trick and I can now resume my simulation, thanks :)<br><br>Cheers,<br>Jakob<br><br><div class="gmail_quote">2011/2/18 Oleg Korobkin <span dir="ltr"><<a href="mailto:korobkin@phys.lsu.edu" target="_blank">korobkin@phys.lsu.edu</a>></span><br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Hi Jakob,<br>
<br>
I had the same problem some time ago. Since your HDF5 files are<br>
compressed, Cactus tries to unpack them before reading, and this<br>
operation somehow is very memory-intensive. Try to repack your<br>
checkpoint files before restarting to reduce compression level to zero.<br>
You can use the standard h5repack utility for that. Here's a small<br>
script to uncompress all checkpoint files and put them to an $OUTDIR<br>
directory:<br>
<br>
----------------<br>
#!/bin/bash<br>
<br>
ITER=5555 # which iteration to use<br>
OUTDIR=$SCRATCH/tmp # output directory<br>
for f in checkpoint.chkpt.it_${ITER}.file_*.h5; do<br>
echo processing $f...<br>
h5repack -i $f -o $OUTDIR/$f -f NONE<br>
done<br>
----------------<br>
<br>
This will make your checkpoint files 5-10 times larger, but because they<br>
don't need to be unpacked in RAM, less memory is required to read them.<br>
<br>
Also, try the following:<br>
1. set:<br>
CarpetIOHDF5::open_one_input_file_at_a_time = yes<br>
IO::abort_on_io_errors = yes<br>
2. use less cores per node.<br>
3. try running on larger number of processors.<br>
<br>
Cheers,<br>
- Oleg Korobkin<br>
<div><div></div><div><br>
Frank Loeffler wrote:<br>
> Hi,<br>
><br>
> On Thu, Feb 17, 2011 at 03:24:04PM +0900, Jakob Hansen wrote:<br>
>> #005: H5Zdeflate.c line 133 in H5Z_filter_deflate(): memory allocation<br>
>> failed for deflate uncompression<br>
>> major: Resource unavailable<br>
>> minor: No space available for allocation<br>
><br>
>> Any ideas for possible cause and solution to this?<br>
><br>
> Looking at these error messages I suggest you first do an h5ls/h5dump to<br>
> see if the files are actually ok. If that is so, what I would suspect next<br>
> is that reading the files takes more memory than is available, as indicated<br>
> by the message above. In this case I suggest to try the workarounds<br>
> mentioned in this thread:<br>
><br>
> <a href="http://lists.einsteintoolkit.org/pipermail/users/2011-February/000852.html" target="_blank">http://lists.einsteintoolkit.org/pipermail/users/2011-February/000852.html</a><br>
><br>
> If this also doesn't help I would suggest to try again using more<br>
> available memory, e.g. using more nodes.<br>
><br>
> Frank<br>
><br>
><br>
><br>
</div></div>> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
> <a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
</blockquote></div><br><br clear="all"><br>