[Users] Merging carpet hdf5 data without creating additional files

[Frank Loeffler]

On Tue, May 17, 2016 at 04:52:10PM +0000, Michael Clark wrote:
> Hello, I'm interested in ways to manipulate carpet hdf5 data from multiple
> jobs (e.g. from different output-XXXX directories of a simfactory-managed
> simulation) for visualization purposes.
> 
> Currently, if I had a variable I wanted to plot, what I would do is run
> hdf5_merge on the separate .h5 files to generate a single hdf5 file and
> then use a visualization tool (like visit) with it.
> 
> I'm curious if there is a means to do this differently: for instance, if
> one could load the separate .h5 files in python as some collection of data
> objects and then combine those objects, no separate and combined .h5 file
> would need to be produced.  Then, if some means existed to create a
> visualization from that data object (e.g. through yt), the need for the
> combined .h5 file would be totally obsolete.
> 
> I suspect this must already be possible in some manner, perhaps using other
> programs.  I'd be curious to hear how, for instance, SimulationTools
> handles this problem.

Try the attached script. It uses python to get a list of datasets from
however many hdf5 files you specify and creates a new (small) hdf5 file
that contains only links to the original files. This can be used to,
e.g., only specify a single file in VisIt, but can also be used (and is,
if you look into the script), to 'filter' some datasets (timesteps,
refinement levels, whatever you like) which VisIt wouldn't see, and
thus, not initially load, reducing load times by a lot in some cases.

The primary use, however, and the reason I wrote this, was that this way
you can combine variables that had been written to multiple files to
one, allowing much easier manipulation in VisIt, e.g., for vector
components like the hydro velocity. It essentially provides a different
'view' of the same simulation to visit.

Note that because of that filtering, you might want to have a look at
the (very short) script and adapt it to your needs before you use it.

Also, in case you have an old version of VisIt, you might need to enable
the use of soft links in the VisIt Carpet reader. I am currently not
sure if that change made it back upstream yet. The corresponding patch
is:

Index: avtCarpetHDF5FileFormat.C
===================================================================
--- avtCarpetHDF5FileFormat.C   (revision 57)
+++ avtCarpetHDF5FileFormat.C   (working copy)
@@ -1044,7 +1044,7 @@
    sprintf(fullname, "%s%s%s", rootname, rootname[strlen(rootname)-1]=='/' ? "" : "/",member_name);
    
    // we are interested in datasets only - skip anything else
-   H5Gget_objinfo (group_id, member_name, 0, &object_info);
+   H5Gget_objinfo (group_id, member_name, 1, &object_info);
    if (object_info.type != H5G_DATASET)
    {
       if (object_info.type == H5G_GROUP)


Frank

-------------- next part --------------
A non-text attachment was scrubbed...
Name: hdf5_merge.py
Type: text/x-python
Size: 875 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20160517/3d663419/attachment.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20160517/3d663419/attachment.bin