#114: per-variable tolerances for Cactus testsuites --------------------------+------------------------------------------------- Reporter: knarf | Owner: tbode Type: enhancement | Status: review Priority: major | Milestone: ET_2011_11 Component: Cactus | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by tbode):
Thanks for looking at the patch.
* Concerning debugging code, the only debugging left uncommented is in PrintDataBase, which is in there specifically ''for'' debugging and only for debugging. This should be left in there so that the database printing routine descends one step lower in the hierarchy, giving useful information instead of just the reference to the hash. There is a PrintDataBase call in RunTest.pl that is commented out. If we don't want any commented debugging, this should also be deleted. If we want no debugging at all committed, that entire function should be removed. * I fully expected a complaint concerning the keywords. Easily changed. Long all-caps really annoy me so I prefer not to use it while coding. Search and replace are sufficient. I apologize for having left the case is is. Figured there would be enough discussion concerning functionality that there would be changes anyway. I didn't think of a better choice for the keyword, though, that wasn't absurdly long or misleading. * I implemented it as an extra keyword as it was easier to parse the lines this way, and specifying the expression first and tolerance second makes more sense to me as a user. Changing how ABSTOL and RELTOL are specified would outdate all current test.ccl files and would be a much more invasive change. I could put in extra loops of logic to treat both old and new cases. I should think on this some more, whether there is a good way to handle this, but I prefer a second keyword.
The interface in the patch ''does'' allow any number of expressions to be specified at either level. Where it falls short is in logic when a file matches more than one expression. Currently any test tolerances override thorn-wide tolerances, and expression-specific tolerances blindly override all other tolerances. If it matches more than one expression, though, it's only the last match's tolerance that is used. How should we treat cases where a filename matches more than one expression? I'm inclining towards throwing an error if this happens indicating that test.ccl should be modified.