[Users] Fwd: [Hdf-forum] hdf-1.10 and collective metadata success story

[Erik Schnetter]

As many of you know, restarting a simulation from checkpoints is often
slower than it should be, and we have in the past implemented several
performance improvements and work-arounds to address this. However, this is
still an unsolved problem. The performance of accessing HDF5 datasets (i.e.
opening the datasets and/or reading attributes) was often substantially
slower than expected. The performance of reading the data themselves seems
to be fine.

A few weeks ago, HDF5 1.10 was released. This features (optionally) a new
file format that stores metadata (i.e. group directories and attributes) in
a more efficient way. I always suspected that this should help with
performance, but didn't run benchmarks at scale yet. Today, I see this
email from Rob Latham <http://www.mcs.anl.gov/~robl/> on the HDF5 mailing
list, which I copy below.

Essentially: HDF5 1.10 can provide a significant performance increase, and
is worth a try.

If you try this at home, remember that HDF5 1.10 can create files that are
backward compatible with 1.8. Obviously, if you do this, you will not see
the performance benefits of the new file format. Please read the release
notes to learn more about this, and about backward compatibility. Note that
HDF5 1.10 also ships with a tool "h5format_convert" that converts from the
1.10 to the 1.8 format in case you have tools that can't read the new
format. This conversion is very quick.

-erik


---------- Forwarded message ----------
From: Rob Latham <robl at mcs.anl.gov>
Date: Tue, May 24, 2016 at 10:44 AM
Subject: [Hdf-forum] hdf-1.10 and collective metadata success story
To: hdf-forum at lists.hdfgroup.org


Since I've spent oh, only the last 8 years beating you up a bit about HDF5
metadata reads at scale, it is only fair that I share with you some good
news.  Thanks for implementing a scalable metadata read.  It's already
paying off.

There was a user trying to read a large dataset on our Blue Gene.  We got
him using 1.10, and presto:

Here's a quote from the user:

Yes, I can now restore data (with hdf5 1.10).  I did a large test
simulation on cetus (with fewer nodes than I'd usually run, so a lot of
data per node), and then followed up by restoring a real 1024-node
simulation on mira: this read 6 "field files" of 3.8 GB each and 2
"particle files" of 396 GB and 2 "particle files" of 69 GB in 442 seconds.
Previously it was taking on the order of an hour to read in a 4 GB field
file.

==rob

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum at lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5



-- 
Erik Schnetter <schnetter at gmail.com>
http://www.perimeterinstitute.ca/personal/eschnetter/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20160524/51d63d34/attachment.html