[ET Trac] [Einstein Toolkit] #808: testsuites in HDF5

[Einstein Toolkit]

#808: testsuites in HDF5
--------------------------+-------------------------------------------------
  Reporter:  jtao         |       Owner:                     
      Type:  enhancement  |      Status:  new                
  Priority:  optional     |   Milestone:  Cactus_4.1.0       
 Component:  Cactus       |     Version:  development version
Resolution:               |    Keywords:                     
--------------------------+-------------------------------------------------

Comment (by hinder):

 This ticket has raised several issues:

 1. The original summary contains an error: Carpet HDF5 output is not
 independent of the number of processes.

 2. Having an output format for test suites which is independent of the
 number of processes would be extremely useful (it makes testing
 correctness on different numbers of processes trivial).  There might be
 ways to set the parameters so that such an ASCII file is created.  Roland:
 I believe that I fixed the issue with the existence of comment/blank lines
 (https://trac.einsteintoolkit.org/browser/Cactus%20flesh/trunk/lib/sbin/RunTestUtils.pl?rev=4773).

 3. It might be possible for Carpet to merge together the different
 components in its "sliced output" routines by super-region.  If this was
 implemented, presumably it would apply to both HDF5 and ASCII "sliced"
 output.  This would make the output independent of the number of
 processes. This might also make Carpet output (at least unigrid) usable
 with generic tools without needing to write plugins.

 4. Using HDF5 instead of ASCII for test output has only one advantage that
 I can see, which is that the data can be transparently compressed so that
 it takes up less space, but it has many disadvantages (note that I am
 usually a rabid supporter of HDF5 as opposed to ASCII).  For example,
 version control systems and their GUIs do not natively understand HDF5
 files, which means that you cannot tell from one version of a file to the
 next what has changed.  For these tests, this sort of comparison is
 essential, and to encourage people to do it, it must be very easy.  Having
 to run an external tool to see the differences adds a lot of unnecessary
 work.

 5. Modifying the Cactus test mechanism to understand HDF5 files is
 something I have thought about for a while, before coming to the
 conclusion in (4).  There is already h5diff
 (http://www.hdfgroup.org/HDF5/doc/RM/Tools.html), which is a standard part
 of HDF5.  Hooking into this to show differences would probably be
 possible.  You can specify relative and absolute tolerances on the command
 line, as well as options for NaN detection.

 6. Should we be using 3D output for tests?  The resulting size, especially
 with Carpet's 3D output format, makes HDF5 the only reasonable option.  I
 have in the past experimented with this, and found that the resulting
 files, even with full HDF5 compression, were too large.  If we had
 comprehensive coverage with 3D output, we would be forced to store them in
 a separate repository.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/808#comment:6>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit