[ET Trac] #2543: Consolidate data formats to simplify postprocessing

[Erik Schnetter]

#2543: Consolidate data formats to simplify postprocessing

 Reporter: Wolfgang Kastaun
   Status: new
Milestone: 
  Version: development version
     Type: enhancement
 Priority: minor
Component: 

Comment (by Erik Schnetter):

The key `format_name` and `format_version` tell you how to interpret the content of the file. I don’t think it is feasible to have a generic description in the metadata that would allow you to extract the information from all the file formats we support. \(If that was possible then we wouldn’t have multiple file formats.\) Thus the reader needs to understand and have special support for the specific file format, be it Silo or HDF5 or TSV or JSON.

In this particular case, the TSV file has column headers that identify the content. The TSV files are small, and reading them is fast.

Regarding slow reading: In the example I provided, there are two files written per iteration, one TSV file with the norms, and one Silo/HDF5 file with 3D data. If there had been many nodes, then there might have been more files to allow nodes to write data independently \(which speeds things up\).

If there is a thorn that produces many files, then the thorn should be updated, or it should switch to a standard output format. That issue is independent of setting up a table of contents. There are probably cases where having many ASCII files makes sense, e.g. for debugging or developing scripts.

I chose yaml since that is already supported in CarpetX, and because it is human-readable. The file encoding really isn’t important and can be changed quickly.

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2543/consolidate-data-formats-to-simplify
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/trac/attachments/20210803/76cd5d3b/attachment.html