Hi,
We had quite a discussion about the problems the current visualization tools have reading our hdf5 format. It became clear that we have to change/extend it, but before we think about how, I would like to see what specifically the readers have problems with. Let me start here with what I can recall, and please freely add to that.
One of the problems is that readers currently have to iterate over all datasets present to get to some information they need. Getting the list of all datasets by itself can take quite a while on a regular hdf5 file, but then readers also have to look at attributed within each dataset. While of of this is necessary to visualize all data in a given file, most of the time not all of the data is actually necessary, and certainly not for just 'opening the file'.
operations a reader needs to be fast: - list of variables (at a given iteration) - list of time steps / iterations - AMR structure for one given iteration (all maps, rls and components)
Regardless of these additional meta-data, we already established the need for a meta-data file, effectively a copy of all datasets but without the actual data. Am I remembering correctly that the idea was to write this at run-time (eventually - right now it could be generated as post-processing)?
Is this all readers would need?
Frank