[ET Trac] #2543: Consolidate data formats to simplify postprocessing

[Erik Schnetter]

#2543: Consolidate data formats to simplify postprocessing

 Reporter: Wolfgang Kastaun
   Status: new
Milestone: 
  Version: development version
     Type: enhancement
 Priority: minor
Component: 

Comment (by Erik Schnetter):

The idea is to have one such metafile per output directory. I’ll have to think about the one-file-per-iteration setup, it does seem wasteful and inconvenient. There certainly won’t be one file per node, things would be aggregated.

If you like the design for CarpetX, then we can repeat it for Carpet \(or, rather, CactusBase/IOUtil\) and use it for all simulations. I’m sure we’ll have to iterate on the design, and then flush out a few bugs where the design doesn’t make sense or is too limited.

0D output etc. are already supported in the format. There is a key that specifies which directions of a variable is output: `[0,1,2]` is for 3D output, `[1]` is for output in the y direction, etc.

Yes, HDF5 is slow. That’s why I want to switch to ADIOS2. This format has essentially the same capabilities as HDF5 \(blocks of data, arbitrary types, attributes, groups, etc.\), but it properly separates metadata, and is parallel by design. It’s also safe to append to an ADIOS2 file \(internally data are separated into “iterations” which cannot be modified once written\).

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2543/consolidate-data-formats-to-simplify
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/trac/attachments/20210803/c1b11940/attachment.html