#1483: speed up reading metadata of HDF5 files in visit --------------------------+------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: minor | Milestone: Component: Other | Version: development version Resolution: | Keywords: CarpetHDF5 --------------------------+-------------------------------------------------
Comment (by rhaas):
I made up some test using a file with very many (~400k) datasets (9.2GB which does not quite fit into main memory). I give times to iterate over the file and read the metadata (using gettimeofday):
With patch and without index files: 26s With patch and with index files: 18s Without patch and without index files: 27s Without patch and with index files: 19s
Doing the same with a large 3d dataset (~16GB) I find:
With patch and without index files: 12s With patch and with index files: 10s Without patch and without index files: 12s Without patch and with index files: 10s
So ti seems as if this has not measurable impact in speed (index files however help). This would mean that there is no need (anymore?) to try and parse information out of the dataset name and one can just query the attributes.