[ET Trac] [Einstein Toolkit] #1872: test system does not detect extra lines in output files

[Einstein Toolkit]

#1872: test system does not detect extra lines in output files
-----------------------+----------------------------------------------------
 Reporter:  rhaas      |       Owner:                     
     Type:  defect     |      Status:  new                
 Priority:  critical   |   Milestone:  ET_2016_11         
Component:  Cactus     |     Version:  development version
 Keywords:  testsuite  |  
-----------------------+----------------------------------------------------
 I accidentally removed some lines of output from the stored test suite
 data in git hash
 [https://bitbucket.org/eloisa/ctthorns/commits/79c5e0d32fb1e3d6b219d7b55211fa3c388f7268
 79c5e0d32fb1e3d6b219d7b55211fa3c388f7268] of ctthorns in the files
 {{{CT_MultiLevel/test/constraints_spherical/*_norm_eqn?.asc}}}. Yet the
 test all passed. I have since restored the changed files.

 This I would consider quite serious since extra output lines should always
 cause the test to fail rather than being silently ignored.

 I am marking this as critical since it potentially renders the test suites
 useless when detecting changes. If we fell we can "document away" this
 then, it can be downgraded to "major".

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1872>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit