Hello all,
The failing tests are
GRHydro.GRHydro_test_shock_weno NaNChecker.nancount
These started failing when we moved the test VM from one host to another, so this is probably CPU-dependent.
NaNCount is odd since the code is new enough that it should just set up nans directly and the cfg file does not set -ffast-math so NaNs should be detected. I will try and see if I can reproduce this on my laptop but it seems this is very specific to the VMs used for Jenkins.
Note that the many failures on comet and gordon seem to be related to file corruption when producing ASCII Carpet output (though CarpetIOASCII seems bug free). The corruption goes away if I force flushing of the buffers after each line (using std::endl instead of "\n"). This happens on both comet and gordon which use different versions of the intel compiler (2015 and 2013) and different gcc compilers (for the C++ library). It almost sounds like a file system issue (they both use similar scratch file systems since they are both at SDSC) and the issue goes away if I run in $HOME rather than $SCRATCH.
Yours, Roland