<div dir="ltr">On Tue, Aug 18, 2015 at 4:02 PM, Frank Loeffler <span dir="ltr"><<a href="mailto:knarf@cct.lsu.edu" target="_blank">knarf@cct.lsu.edu</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
We had quite a discussion about the problems the current visualization<br>
tools have reading our hdf5 format. It became clear that we have to<br>
change/extend it, but before we think about how, I would like to see<br>
what specifically the readers have problems with. Let me start here with<br>
what I can recall, and please freely add to that.<br>
<br>
One of the problems is that readers currently have to iterate over all<br>
datasets present to get to some information they need. Getting the list<br>
of all datasets by itself can take quite a while on a regular hdf5 file,<br>
but then readers also have to look at attributed within each dataset.<br>
While of of this is necessary to visualize all data in a given file,<br>
most of the time not all of the data is actually necessary, and<br>
certainly not for just 'opening the file'.<br>
<br>
operations a reader needs to be fast:<br>
- list of variables (at a given iteration)<br>
- list of time steps / iterations<br>
- AMR structure for one given iteration (all maps, rls and components)<br>
<br>
Regardless of these additional meta-data, we already established the<br>
need for a meta-data file, effectively a copy of all datasets but<br>
without the actual data. Am I remembering correctly that the idea was to<br>
write this at run-time (eventually - right now it could be generated as<br>
post-processing)?<br>
<br>
Is this all readers would need?<br></blockquote><div><br></div><div>Frank</div><div><br></div><div>As Jonah described during the ET workshop, we're working with the yt developers to make Carpet's HDF5 format yt-friendly. At the moment, we are adding the missing information, which so far included the set of active grid points -- i.e. those that should be displayed for a given level, as opposed to those that should be cut off (ghost, buffer, symmetry points). One may argue that these points should not have been output at all, but that would be a major change to the current file format.</div><div><br></div><div>Additional information may also be useful. This is information that is already present in some way, but is cumbersome to extract. Presumably, Carpet already reconstructs this information when reading data from a file, but a visualization or analysis tools may want to access information in a different order, and may not want to reimplement most of CarpetIOHDF5.</div><div><br></div><div>The items you list are a good starting point, but are too high-level to be useful as a guide to implementing this. To make things concrete, I'd rather collaborate with someone who is actually implementing a reader, and provide the data that this reader needs. For example, you say you want a "list of variables (as a given iteration)" -- do you really want a two-step table, containing first a list of iterations, and then (for each iteration) a list of variables? That's likely very different from what a user wants to extract; rather, people want a set of variables, and for each variable, the set of iterations at which this variable has data. Given that we have an AMR format where certain levels exist only at certain iterations, this requires a bit more detail for a complete specification.</div><div><br></div><div>How do you want the AMR structure to be presented? Currently, Carpet can output a string that can be parsed reasonably easily, and which describes the grid structure. Is that sufficient?</div><div><br></div><div>I hear the visualization tools are interested in learning the connectivity between the different refinement levels. How should this be represented? With vertex centering, the vertices overlap, and if you want to display the cells around each vertex (control volume, necessary for volume rendering), then you need to handle partial volumes. With cell centering, things are easier, at least if you use CarpetRegrid2::snap_to_coarse -- which Carpet doesn't require, but the visualization tool may want to insist on it.</div><div><br></div><div>So, in addition to giving a high-level list of features, let's also collect a list of tools that will help us find out the details about these wanted features.</div><div><br></div><div>We also may want to have a mechanism to "glue" the different output file from different processors together, other than just looking for files with similar names.</div><div><br></div><div>Finally, I disagree with the "established the need for a meta-data file". This capability exists, and it speeds up reading for the current output format, but the current output format has several obvious shortcomings; if those were remedied, things may be much faster.</div><div><br></div><div>-erik</div></div><div><br></div>-- <br><div class="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
</div></div>