<div dir="ltr">On Wed, Aug 19, 2015 at 12:46 PM, Frank Loeffler <span dir="ltr"><<a href="mailto:knarf@cct.lsu.edu" target="_blank">knarf@cct.lsu.edu</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I am not a visualization expert, so I am only answering what I know.<br>
Please others: chime in. I only wanted to get the discussion going and<br>
avoid different people working on the same thing separately after the<br>
workshop.<br>
<span class=""><br>
On Wed, Aug 19, 2015 at 11:51:00AM -0400, Erik Schnetter wrote:<br>
> As Jonah described during the ET workshop, we're working with the yt<br>
> developers to make Carpet's HDF5 format yt-friendly. At the moment, we are<br>
> adding the missing information,<br>
<br>
</span>Great. It might be good to share this with others.<br>
<span class=""><br>
> which so far included the set of active<br>
> grid points -- i.e. those that should be displayed for a given level, as<br>
> opposed to those that should be cut off (ghost, buffer, symmetry points).<br>
> One may argue that these points should not have been output at all, but<br>
> that would be a major change to the current file format.<br>
<br>
</span>We already have parameters to do that.<br>
<span class=""><br>
> The items you list are a good starting point, but are too high-level to be<br>
> useful as a guide to implementing this. To make things concrete, I'd rather<br>
> collaborate with someone who is actually implementing a reader, and provide<br>
> the data that this reader needs. For example, you say you want a "list of<br>
> variables (as a given iteration)" -- do you really want a two-step table,<br>
> containing first a list of iterations, and then (for each iteration) a list<br>
> of variables?<br>
<br>
</span>That is the kind of discussion I wanted to get started.<br>
<span class=""><br>
> That's likely very different from what a user wants to<br>
> extract; rather, people want a set of variables, and for each variable, the<br>
> set of iterations at which this variable has data.<br>
<br>
</span>I agree.<br>
<span class=""><br>
> How do you want the AMR structure to be presented? Currently, Carpet can<br>
> output a string that can be parsed reasonably easily, and which describes<br>
> the grid structure. Is that sufficient?<br>
<br>
</span>I didn't try this myself, but I remember someone at the workshop<br>
mentioning parsing the string as 'quite complicated' (no quote). Does<br>
this string also include information about all the components, so that a<br>
reader can easily figure out which component is containing a specific<br>
point/region?<br></blockquote><div><br></div><div>The yt developers -- who probably have more of a computer science background -- were not bothered by the format.</div><div><br></div><div>Feel free to suggest a different format, or a different mechanism. The grid structure consists essentially of two nested arrays: A set of refinement levels, and each has a set of components. A component is described by the location of the origin and its size, i.e. two integer three-vectors. In C++, you would have something like</div><div><br></div><div>struct bbox { int origin[3], size[3]; };</div><div>typedef vector<vector<bbox>> grid_structure;</div><div><br></div><div>There's a slight complication if you have a multi-block system (adding either another level to the arrays, or adding an integer to the bbox structure describing the patch number). You also need to be careful about how the integer coordinates between the refinement levels are related, which is different for vertex and cell centred grids.</div><div><br></div><div>One way to store this without using a hierarchy would be to define instead</div><div><br></div><div>struct bbox {</div><div> int reflevel;</div><div> int patch;</div><div> int origin[3];<br></div><div> int size[3];</div><div>};</div><div><br></div><div>and then store all such bboxes in a single, large array. It is the up to the reader to split this into a tree structure according to the reflevel and patch entries. That's easier to read, but more difficult to process.</div><div><br></div><div>-erik</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> We also may want to have a mechanism to "glue" the different output file<br>
> from different processors together, other than just looking for files with<br>
> similar names.<br>
<br>
</span>I experimented with external links in hdf5. They work rather nicely<br>
(after changing one byte in the source of the VisIt reader: enabling<br>
following links). I use this to have one (very small) hdf5 file pointing<br>
to all three components of a vector, so that it is easy to combine them<br>
to a vector in VisIt (since then from the point of visit they come from<br>
the same database). The 'recombiner' for this is a very tiny python<br>
script, but this could be done during a simulation as well.<br>
<span class=""><br>
> Finally, I disagree with the "established the need for a meta-data file".<br>
> This capability exists, and it speeds up reading for the current output<br>
> format, but the current output format has several obvious shortcomings; if<br>
> those were remedied, things may be much faster.<br>
<br>
</span>The main idea this reasoning comes from is that if meta-data and data<br>
are written intermixed, then just reading the meta-data is always going<br>
to be slower than if it would be contained in a file missing the actual<br>
data, due to disk access usually reading much more than requested. This<br>
is separate from what the meta data actually looks like.<br>
<span class="HOEnZb"><font color="#888888"><br>
Frank<br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
</div></div>