As many of you know, restarting a simulation from checkpoints is often slower than it should be, and we have in the past implemented several performance improvements and work-arounds to address this. However, this is still an unsolved problem. The performance of accessing HDF5 datasets (i.e. opening the datasets and/or reading attributes) was often substantially slower than expected. The performance of reading the data themselves seems to be fine.
A few weeks ago, HDF5 1.10 was released. This features (optionally) a new file format that stores metadata (i.e. group directories and attributes) in a more efficient way. I always suspected that this should help with performance, but didn't run benchmarks at scale yet. Today, I see this email from Rob Latham http://www.mcs.anl.gov/~robl/ on the HDF5 mailing list, which I copy below.
Essentially: HDF5 1.10 can provide a significant performance increase, and is worth a try.
If you try this at home, remember that HDF5 1.10 can create files that are backward compatible with 1.8. Obviously, if you do this, you will not see the performance benefits of the new file format. Please read the release notes to learn more about this, and about backward compatibility. Note that HDF5 1.10 also ships with a tool "h5format_convert" that converts from the 1.10 to the 1.8 format in case you have tools that can't read the new format. This conversion is very quick.
-erik
---------- Forwarded message ---------- From: Rob Latham robl@mcs.anl.gov Date: Tue, May 24, 2016 at 10:44 AM Subject: [Hdf-forum] hdf-1.10 and collective metadata success story To: hdf-forum@lists.hdfgroup.org
Since I've spent oh, only the last 8 years beating you up a bit about HDF5 metadata reads at scale, it is only fair that I share with you some good news. Thanks for implementing a scalable metadata read. It's already paying off.
There was a user trying to read a large dataset on our Blue Gene. We got him using 1.10, and presto:
Here's a quote from the user:
Yes, I can now restore data (with hdf5 1.10). I did a large test simulation on cetus (with fewer nodes than I'd usually run, so a lot of data per node), and then followed up by restoring a real 1024-node simulation on mira: this read 6 "field files" of 3.8 GB each and 2 "particle files" of 396 GB and 2 "particle files" of 69 GB in 442 seconds. Previously it was taking on the order of an hour to read in a 4 GB field file.
==rob
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5