[Users] visualization-friendly HDF5 format

[Frank Loeffler]

Hi

On Thu, Aug 20, 2015 at 09:49:52AM +0200, Roland Haas wrote:
> A good compromise would be a (it seems to me) a hierarchical structure
> that is first indexed by timestep along with a table for times as a
> function of timestep (since the timestep is arbitrary and just a counter
> in Cactus). This would implicitly assume that the grid structure is
> identical between variables

Why would this assume that? In file system terms the first level of
directories would just sort the components into their respective times.

> (which need not be true and in fact I had a
> case where this was not true).

You have an example where the grid structure is different? Is this
because you can specify, say, the refinement levels separately in the
par file for each variable? That's an interesting use-case we should
consider.

> An alternative that may be easier for visualization is to index by
> variable name first.

We can have both. My current idea is to keep the current structure for
backward compatibility, only adding meta data and structure (trees)
which use symlinks to point to the actual components on the root level.
Older readers could still read new files, newer readers could leverage
the new information to be faster, and old files could even be 'upgraded'
by adding the necessary new information.

> I would actually prefer if this information was included in the HDF5
> files itself (replicated in each of them if possible) and written at the
> time the HDF5 file was created. We have some type of metadata in the
> index files that CarpetIOHDF5 can write yet they are not sufficient for
> very large datasets (since they do no reduce the number of files one has
> to parse).

I see the value of that. In order to have this working efficiently, we
could work with pre-allocating larger chunks for meta-data, and filling
them bit by bit. This should (hopefully) make it faster than reading
meta-data all over the output file.

Frank

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20150820/c22624f1/attachment-0001.bin