Hello all,
comet file corruption:
- CarpetIOASCII output files can contain corrupted lines of output in the middle of a component
- happens only on the scratch file system, not on $HOME and only for more than 1 MPI ranks
- happens for both the current release version of ET (ET_PayonGaposhkin 2016_11) as well as old (years old) versions of Carpet
- workaound is to force flushing after each line by using std::eol instead of "\n". This may be slow since flushing can be a very slow operation (milliseconds if it flushes all the way to the physical disk)
- a workaround may be to write output into a string stream first then dump its rdbuf() to file
- will engage SDSC support to see if they have suggestions on how to avoid this issue without having to first write to a local file, then copy to scratch once the run finishes
This seems to be a file system issue. I modified the runscript to sandwich strace into the call (14 in the RunScript ended up being the file descriptor used for ASCII output) and used that information and gawk:
gawk -vFS='"' '/write.*/grid-coordinates.xy.asc/{print "printf ""$2"""}' ~/strace.18826.log >recreate.sh
to produce a shell script that when run, would reply all the write() calls that the MPI code used. Running the script produces a file that differs from what what Cactus produced when run under MPI and which agrees with the data in the testsuite.
So: Cactus calls the OS write() function with the correct data but somehow the OS writes incorrect data to disk.
I will file a ticket with SDSC support.
Yours, Roland