[ET Trac] [Einstein Toolkit] #1565: change default of IOUtil::out_save_parameters to true

[Einstein Toolkit]

#1565: change default of IOUtil::out_save_parameters to true
--------------------------+-------------------------------------------------
  Reporter:  rhaas        |       Owner:                     
      Type:  enhancement  |      Status:  review             
  Priority:  optional     |   Milestone:                     
 Component:  Cactus       |     Version:  development version
Resolution:               |    Keywords:                     
--------------------------+-------------------------------------------------

Comment (by rhaas):

 Wolfgang: the parameters stored in "/Parameters and Global Attributes/All
 Parameters" are those that Cactus detects as having been set (including
 the "setting" of parameters from checkpoints it seems). The attached short
 python script parses the dataset and outputs all information to screen
 (the actual parsing are the 6 lines
 {{{
     pairs = dset[()].splitlines()
     for pair in pairs:
         m = re.match("^([^ ]*) = (.*)$", pair)
         if(not m):
             raise ValueError("invalid paramter setting '%s'" % pair)
         parvals[m.group(1)] = DecodeString(m.group(2))
 }}}
 You can already get the full set of parameters with the current code by
 setting IOUtil::out_save_parameters = "all" explicitly. Making them all
 attributes would likely be possible though I am not sure how much more
 useful that would be. One would then likely want to use scalars int/double
 values for int/real params, int for booleans and some string type for
 strings and keywords. Arrays of parameters should then appear as arrays
 which for the strings requires variable length HDF5 strings which are not
 very simple to handle (in C/C++. Python hides this). Most of the code to
 handle this is likely somewhere in CarpetIOHDF5 but it would require some
 effort to get eg the mapping of CCTK_REAL to double or float based on
 compile time options right.

 Ian: currently IO::parfile_write = "generate" only writes parameters that
 have been set, so it is not any better than the current information in
 hdf5 files. It also writes the parameters only at the beginning (ie before
 anything is steered) which is less useful (though not worse than hdf5
 files which only retain the last set of parameters). There's in principle
 the parameter 'parfile_update_every' which would Cactus regenerate the
 parfile every so often however that is currently not implemented (see
 IOUtil's Startup.c).
 Concerning having "cooked" ie processed from parameters metadata in files,
 there is the issue of only storing information once since otherwise I very
 much suspected that the different pieces of information will contradict
 each other. Maybe not much of an issue for postprocessing but I would try
 to avoid having the actual simulation rely on this postprocessed data. Ie.
 if you need to know if a symmetry is active, you should either consult the
 parameters or call a function to query the symmetry state. One should not,
 I believe query some generic metadata collection for this information
 since it will very likely become out of sync with whatever information was
 initially used to compute it.

 Erik: The idea of including this information in the hdf5 files is to have
 them as self contained as possible. Formaline is certainly the best place
 to collect the metadata since it is designed for this.
 Possibly the flesh could define an API to register the metadata and the
 let formaline take care of generating and saving it. This seems to be the
 general Cactus approach the the flesh only defines an API and then the
 thorns implement it.
 Parsing formaline-jar.txt to get out the information seems to be non-
 trivial (more than 6 lines). In order to detect the end-of-value for
 multi-line string parameter values I seem to need to keep track of double
 quotes to remember when I am inside of a double quoted parameter. The
 format in the hdf5 files where each parameter value is a single line seems
 easier to parse for a machine (and harder for a human).

 This discussion seems to have gotten a little bit off-track though. I was
 really only asking to change the default value for an existing parameter
 from its current default ("only set") to another of its currently
 implemented values ("all") since the current default is not very useful. I
 did not want to enter into the more general discussion of what metadata to
 include and in which format to provide it. What I propose can be
 implemented in 5 minutes (change of default value) and is immediately
 useful, while the current situation ''still'' writes the dataset to
 ''all'' hdf5 files and the dataset is almost never useful.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1565#comment:13>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit