#808: testsuites in HDF5 --------------------------+------------------------------------------------- Reporter: jtao | Owner: Type: enhancement | Status: new Priority: optional | Milestone: Cactus_4.1.0 Component: Cactus | Version: development version Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by hinder):
This ticket has raised several issues:
1. The original summary contains an error: Carpet HDF5 output is not independent of the number of processes.
2. Having an output format for test suites which is independent of the number of processes would be extremely useful (it makes testing correctness on different numbers of processes trivial). There might be ways to set the parameters so that such an ASCII file is created. Roland: I believe that I fixed the issue with the existence of comment/blank lines (https://trac.einsteintoolkit.org/browser/Cactus%20flesh/trunk/lib/sbin/RunTe...).
3. It might be possible for Carpet to merge together the different components in its "sliced output" routines by super-region. If this was implemented, presumably it would apply to both HDF5 and ASCII "sliced" output. This would make the output independent of the number of processes. This might also make Carpet output (at least unigrid) usable with generic tools without needing to write plugins.
4. Using HDF5 instead of ASCII for test output has only one advantage that I can see, which is that the data can be transparently compressed so that it takes up less space, but it has many disadvantages (note that I am usually a rabid supporter of HDF5 as opposed to ASCII). For example, version control systems and their GUIs do not natively understand HDF5 files, which means that you cannot tell from one version of a file to the next what has changed. For these tests, this sort of comparison is essential, and to encourage people to do it, it must be very easy. Having to run an external tool to see the differences adds a lot of unnecessary work.
5. Modifying the Cactus test mechanism to understand HDF5 files is something I have thought about for a while, before coming to the conclusion in (4). There is already h5diff (http://www.hdfgroup.org/HDF5/doc/RM/Tools.html), which is a standard part of HDF5. Hooking into this to show differences would probably be possible. You can specify relative and absolute tolerances on the command line, as well as options for NaN detection.
6. Should we be using 3D output for tests? The resulting size, especially with Carpet's 3D output format, makes HDF5 the only reasonable option. I have in the past experimented with this, and found that the resulting files, even with full HDF5 compression, were too large. If we had comprehensive coverage with 3D output, we would be forced to store them in a separate repository.