Hello Frank, all,
The garbled output, if not coming from two processes, look like a race. If it is, an explicit flush after output wouldn't solve the problem, but instead 'just' make it less likely to happen.
It only happens with 2 MPI ranks, yet I have checked that indeed only a single rank writes data (or opens the file). This happens during a single open/close cycle so it also is not related to opening/closing the file.
Yes. This smells like the filesystem or OS not invalidating a cache after a (probably cached) write to a file happened. The sysadmins should either know about it, or should be interested in it. If this is indeed the case, it should be possible to generate a simple test case show-casing the problem.
"Simple" is relative. The simplest reproducer I have is CarpetIOASCII's newsep test which still fires up a lot of Cactus thorns (not really required to just check that "-" is used instead of "::" for file names). A simple test code that writes the same lines of data does not show the problem (when run using MPI and 2 ranks of which only rank 0 actually writes).
Yours, Roland