#1565: change default of IOUtil::out_save_parameters to true --------------------------+------------------------------------------------- Reporter: rhaas | Owner: Type: enhancement | Status: review Priority: optional | Milestone: Component: Cactus | Version: development version Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by rhaas):
Wolfgang: the parameters stored in "/Parameters and Global Attributes/All Parameters" are those that Cactus detects as having been set (including the "setting" of parameters from checkpoints it seems). The attached short python script parses the dataset and outputs all information to screen (the actual parsing are the 6 lines {{{ pairs = dset[()].splitlines() for pair in pairs: m = re.match("^([^ ]*) = (.*)$", pair) if(not m): raise ValueError("invalid paramter setting '%s'" % pair) parvals[m.group(1)] = DecodeString(m.group(2)) }}} You can already get the full set of parameters with the current code by setting IOUtil::out_save_parameters = "all" explicitly. Making them all attributes would likely be possible though I am not sure how much more useful that would be. One would then likely want to use scalars int/double values for int/real params, int for booleans and some string type for strings and keywords. Arrays of parameters should then appear as arrays which for the strings requires variable length HDF5 strings which are not very simple to handle (in C/C++. Python hides this). Most of the code to handle this is likely somewhere in CarpetIOHDF5 but it would require some effort to get eg the mapping of CCTK_REAL to double or float based on compile time options right.
Ian: currently IO::parfile_write = "generate" only writes parameters that have been set, so it is not any better than the current information in hdf5 files. It also writes the parameters only at the beginning (ie before anything is steered) which is less useful (though not worse than hdf5 files which only retain the last set of parameters). There's in principle the parameter 'parfile_update_every' which would Cactus regenerate the parfile every so often however that is currently not implemented (see IOUtil's Startup.c). Concerning having "cooked" ie processed from parameters metadata in files, there is the issue of only storing information once since otherwise I very much suspected that the different pieces of information will contradict each other. Maybe not much of an issue for postprocessing but I would try to avoid having the actual simulation rely on this postprocessed data. Ie. if you need to know if a symmetry is active, you should either consult the parameters or call a function to query the symmetry state. One should not, I believe query some generic metadata collection for this information since it will very likely become out of sync with whatever information was initially used to compute it.
Erik: The idea of including this information in the hdf5 files is to have them as self contained as possible. Formaline is certainly the best place to collect the metadata since it is designed for this. Possibly the flesh could define an API to register the metadata and the let formaline take care of generating and saving it. This seems to be the general Cactus approach the the flesh only defines an API and then the thorns implement it. Parsing formaline-jar.txt to get out the information seems to be non- trivial (more than 6 lines). In order to detect the end-of-value for multi-line string parameter values I seem to need to keep track of double quotes to remember when I am inside of a double quoted parameter. The format in the hdf5 files where each parameter value is a single line seems easier to parse for a machine (and harder for a human).
This discussion seems to have gotten a little bit off-track though. I was really only asking to change the default value for an existing parameter from its current default ("only set") to another of its currently implemented values ("all") since the current default is not very useful. I did not want to enter into the more general discussion of what metadata to include and in which format to provide it. What I propose can be implemented in 5 minutes (change of default value) and is immediately useful, while the current situation ''still'' writes the dataset to ''all'' hdf5 files and the dataset is almost never useful.