<html>#2543: Consolidate data formats to simplify postprocessing
<table style='border-spacing: 1ex 0pt; '>
<tr><td style='text-align:right'> Reporter:</td><td>Wolfgang Kastaun</td></tr>
<tr><td style='text-align:right'> Status:</td><td>new</td></tr>
<tr><td style='text-align:right'>Milestone:</td><td></td></tr>
<tr><td style='text-align:right'> Version:</td><td>development version</td></tr>
<tr><td style='text-align:right'> Type:</td><td>enhancement</td></tr>
<tr><td style='text-align:right'> Priority:</td><td>minor</td></tr>
<tr><td style='text-align:right'>Component:</td><td></td></tr>
</table>
<p>Currently, writing postprocessing tools for ET is unnecessarily difficult because required information needs to be collected from many locations, has to accommodate competing standards, and sometimes require guesses using heuristics. Below is a list of improvements from the postprocessing viewpoint, which is not complete and can be augmented over time.</p>
<p></p>
<ol>
<li>Table of content for grid variable output. Each output folder should contain a machine-readable file that keeps track of all files containing grid data, with a list of variables and the available timesteps for each variable. Of course this should distinguish between 1D, 2D and 3D output. Currently, one has to open all files and parse the content for metadata, which can be very slow with HDF5. The issue is especially problematic when using one file per group.</li>
<li>The same for reduction output.</li>
<li>A machine readable file with all parameters and their values, including those not set in the parfile and set to default. The values should be values, postprocessing code should not have to emulate the handmade programming language parfiles have become. Each folder with a restart should have one such file in a standard location/name.</li>
<li>The reductions thorn should also output enough information to convert norm1/average into volume integrals, i.e. a scalar x such that <code>volume integral = x * average</code></li>
<li>Unique extensions. There should be one and only one unique extension for each type of file, across all standard thorns that produce output. In particular, just adding ‘.h5’ is not enough. For example 3D data currently has extension <code>xyz.h5</code> or just <code>.h5</code> and multipole data can have extension <code>.h5</code> as well.</li>
<li>Simfactory should also provide machine-readable metadata about restarts and simulation folders. It should be possible to easily obtain a tree-like structure of the various restarts, complete with iteration ranges.</li>
<li>One standard format for timeseries. Currently reducti
<p>--<br/>
Ticket URL: <a href='https://bitbucket.org/einsteintoolkit/tickets/issues/2543/consolidate-data-formats-to-simplify'>https://bitbucket.org/einsteintoolkit/tickets/issues/2543/consolidate-data-formats-to-simplify</a></p>
</html>