[Users] Merging carpet hdf5 data without creating additional files

[David Radice]

Hello Michael,

I wrote a small python library that does exactly what you are looking for. You can use it to read a set of HDF5 and it returns an object that can be queried for grid functions at particular times and/or locations. The code is publicly available here:

	https://bitbucket.org/dradice/scidata

The repository contains a lot of python modules, most written by me early on during my PhD studies (that is to say most are crap), but the ones that might be interesting for you are:

	scidata/carpet/hdf5.py
	scidata/carpet/grid.py

These have been rewritten many times and they are quite efficient at reading large datasets quickly. We have used them to analyze the simulations from (arXiv:1512.00838), which produced ~200 Tb of data scattered through hundreds of thousands h5 files.

You can see the examples/field.py for an example with 2D data, the handling of 1D and 3D data is identical. Once you have read the data, you can plot or analyze it with your favorite tools (e.g. matplotlib, or yt).

The main caveat is that the code is python-2.7, maybe python-2.6, only. I have not yet found the time / motivation to convert everything to python-3.

Best,

David

> On May 17, 2016, at 9:52 AM, Michael Clark <michael.clark at gatech.edu> wrote:
> 
> Hello, I'm interested in ways to manipulate carpet hdf5 data from multiple jobs (e.g. from different output-XXXX directories of a simfactory-managed simulation) for visualization purposes.
> 
> Currently, if I had a variable I wanted to plot, what I would do is run hdf5_merge on the separate .h5 files to generate a single hdf5 file and then use a visualization tool (like visit) with it.
> 
> I'm curious if there is a means to do this differently: for instance, if one could load the separate .h5 files in python as some collection of data objects and then combine those objects, no separate and combined .h5 file would need to be produced.  Then, if some means existed to create a visualization from that data object (e.g. through yt), the need for the combined .h5 file would be totally obsolete.
> 
> I suspect this must already be possible in some manner, perhaps using other programs.  I'd be curious to hear how, for instance, SimulationTools handles this problem.
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20160517/a7f749cf/attachment.bin