On Thu, Dec 15, 2016 at 10:05:54AM -0600, Roland Haas wrote:
Note that the many failures on comet and gordon seem to be related to file corruption when producing ASCII Carpet output (though CarpetIOASCII seems bug free). The corruption goes away if I force flushing of the buffers after each line (using std::endl instead of "\n").
The garbled output, if not coming from two processes, look like a race. If it is, an explicit flush after output wouldn't solve the problem, but instead 'just' make it less likely to happen.
This happens on both comet and gordon which use different versions of the intel compiler (2015 and 2013) and different gcc compilers (for the C++ library). It almost sounds like a file system issue (they both use similar scratch file systems since they are both at SDSC) and the issue goes away if I run in $HOME rather than $SCRATCH.
Yes. This smells like the filesystem or OS not invalidating a cache after a (probably cached) write to a file happened. The sysadmins should either know about it, or should be interested in it. If this is indeed the case, it should be possible to generate a simple test case show-casing the problem.
Frank