[ET Trac] #2543: Consolidate data formats to simplify postprocessing
trac-noreply at einsteintoolkit.org
Tue Aug 3 13:15:43 CDT 2021
#2543: Consolidate data formats to simplify postprocessing
Reporter: Wolfgang Kastaun
Version: development version
Comment (by Wolfgang Kastaun):
The proposed solution for CarpetX would probably simplify the logic which gathers information on the available data a lot. I’m not entirely sure about speed. The reason parsing the hdf5 files takes so long seems a design flaw that requires basically to read the whole file just to get the names of all datasets, combined with the unfortunate choice of having the table of content information only in those names. I also did not get how the solution would look like for runs using many nodes, would there be one file for each node or even MPI process, or is the information collected first on one node?
But in principle, reading a few thousand yaml files should not take that long, we would have to try.
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2543/consolidate-data-formats-to-simplify
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Trac