[ET Trac] [Einstein Toolkit] #114: per-variable tolerances for Cactus testsuites

[Einstein Toolkit]

#114: per-variable tolerances for Cactus testsuites
--------------------------+-------------------------------------------------
  Reporter:  knarf        |       Owner:  tbode     
      Type:  enhancement  |      Status:  review    
  Priority:  major        |   Milestone:  ET_2011_10
 Component:  Cactus       |     Version:            
Resolution:               |    Keywords:            
--------------------------+-------------------------------------------------

Comment (by tbode):

 It would not be difficult at all to take only the first and last instance,
 however I disagree that uniqueness requires unreadable expressions.  You
 might have to break it up into several lines that are more specific to
 address subsets of the larger group. In the extreme case you would simply
 enter the entire file name. It's certainly possible to make it both unique
 and readable with a minimal familiarity of regular expressions.

 The problem with taking the first or last instance is if the thorn
 developer adds an expression which matches more than was intended,
 increasing the tolerance of a secondary data file to the point where it's
 no longer a good test. There's no verboseness option available in test.ccl
 to developers.  Any warnings put into the RunTest scripts to alert the
 developer would warn everybody, every time they run the test.

 If you insist going with first or last instance, then I would also insist
 we include an option to print a tolerance table for a test, where the
 tolerances for every available datafile in the test are tabulated.  That
 way a developer can assess the success of their expression set without
 having to modify the RunTest*.pl perl scripts.  For some tests this would
 apparently require 200+ lines .... might not actually be a bad addition
 either way actually.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/114#comment:9>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit