On 13 Jul 2017, at 10:48, Miguel Zilhão <mzilhao@ffn.ub.es> wrote:

hi Ian,

NewsB_scri.L02Mm01.asc: substantial differences
      significant differences on 1 (out of 2) lines
      maximum absolute difference in column 1 is 963
      maximum absolute difference in column 2 is 0.000185770963653907
      maximum absolute difference in column 3 is 0.000142466608463344
      maximum relative difference in column 1 is 1
      maximum relative difference in column 2 is 1
      maximum relative difference in column 3 is 1
      ...
The third, SphericalHarmonicReconGen.SpEC-h5-test, fails like this:
NewsB_scri.L02Mm01.asc: substantial differences
      significant differences on 1 (out of 2) lines
      maximum absolute difference in column 1 is 963
      maximum absolute difference in column 2 is 0.000185770963653907
      maximum absolute difference in column 3 is 0.000142466608463344
      maximum relative difference in column 1 is 1
      maximum relative difference in column 2 is 1
      maximum relative difference in column 3 is 1
      ...
I suspect the second and third failures have the same cause. Do we have any idea why these tests fail? They don't seem to fail on any other machines (http://einsteintoolkit.org/testsuite_results/index.php).

i'm not sure whether it's related, but i've had a similar issue with a testsuite of a local thorn i have, which was passing just fine with 1 proc but not with 2 procs (and similar errors). the root of the problem turned out to be that on this particular machine, for whatever reason, when running the parfile with 2 procs the output was "doubled". as if each processor was writing the same thing on the same file. the output itself was the same, but the files that were written were obviously not. so diff was signalling differences in the files where in fact the numbers themselves were (nearly) the same. could you be seeing something like this as well?

Hi,

This is in general a problem; Cactus output is not guaranteed to be the same when you change the number of processes. There are ways to work around this in specific cases, which are used for the tests. Specifically, you can use the options

CarpetIOASCII::compact_format = yes

CarpetIOASCII::output_ghost_points = no

and look at output only in the z direction. If Carpet splits the grid in the z direction, then I think that for small numbers of processes at least, you are guaranteed that the output will be independent of the number of processes, as the processes will write in component order to the file, and the component ordering is the same as the z-ordering in this case.

In your case, you are probably seeing duplicate points because some points (the ghost points) exist on more than one process, and each process outputs all the points it has. Using output_ghost_points = no might help in that case.

For SphericalHarmonicRecon, the tests use PUGH, not Carpet, and compact_format is not available. I will have to check if this is a diffing issue, but I would be surprised, since it works on other machines on 2 processes.