#114: per-variable tolerances for Cactus testsuites --------------------------+------------------------------------------------- Reporter: knarf | Owner: tbode Type: enhancement | Status: review Priority: major | Milestone: ET_2011_10 Component: Cactus | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by tbode):
It would not be difficult at all to take only the first and last instance, however I disagree that uniqueness requires unreadable expressions. You might have to break it up into several lines that are more specific to address subsets of the larger group. In the extreme case you would simply enter the entire file name. It's certainly possible to make it both unique and readable with a minimal familiarity of regular expressions.
The problem with taking the first or last instance is if the thorn developer adds an expression which matches more than was intended, increasing the tolerance of a secondary data file to the point where it's no longer a good test. There's no verboseness option available in test.ccl to developers. Any warnings put into the RunTest scripts to alert the developer would warn everybody, every time they run the test.
If you insist going with first or last instance, then I would also insist we include an option to print a tolerance table for a test, where the tolerances for every available datafile in the test are tabulated. That way a developer can assess the success of their expression set without having to modify the RunTest*.pl perl scripts. For some tests this would apparently require 200+ lines .... might not actually be a bad addition either way actually.