On Mon, May 23, 2016 at 03:35:16PM -0500, Frank Loeffler wrote:
We do have Jenkins failures for a number of tests in QuasiLocalMeasures.
I did look into those, and they turn out to be tricky.
First of all: they are about files that are process-dependent, are generated on one process, but are used by Jenkins also with two processes. They should fail. In the past they didn't, and I believe this is why: https://trac.einsteintoolkit.org/ticket/1889. In short: because of the way absolute/relative errors are currently computed, and because some of the values in this particular case (weights) are 0, comparing the testsuites gives the "How did I get here" error, but this doesn't actually trigger a testsuite failure.
With current flesh, however, the fact that the two-process file does contain more lines, you now still get a strong failure, which made the suite fail now (and Jenkins was correct after all).
The suites pass when run on one process, so the actual data seem to be just fine.
What's left is:
- decide what to do with the testsuites. We could either restrict them to one process only, change the output format to be process-independent, or remove the output of the carpet weight. If I don't get recommendations, I will go ahead within the next days and implement one of these options - likely the second option if easy. - fix the testsuite mechanism. I didn't look closely at #1889, but that might already do that. These differences should have caused a strong failure in the first place.
Frank