#2073: IO corruption on SDSC oasis file systems -------------------------------+-------------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Other | Version: development version Keywords: Comet Gordon SDSC | -------------------------------+-------------------------------------------- I am experiencing file corruption in ASCII output produced by the Cactus code on Comet's (and Gordon's as far as I remember) scratch file systems. This manifests as lines of output being mashed together in the output file.
I have added strace calls to my job script to capture all arguments to the OS's write() function and re-created the write() calls based on this. Those write calls, when replayed on a login node, do produce a correct (no mashed lines) file.
All output to the file in question was from rank 0 only even though the code used MPI and ran on two MPI ranks.
The same code and number of MPI ranks produces a correct output file when run on the $HOME file system.
Thus it seems to me as if there may be an issue with the file system. I can try and reduce the test case to a more minimal example (right now it is a full simulation even though it runs only for <1minute) .
You can find the job script (for account, SLURM options etc) here:
/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/SIMFACTORY/SubmitScript
the script that launches the MPI executable here:
/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/SIMFACTORY/RunScript
the strace output here:
/home/rhaas/strace/strace.1882[67].log
and the awk script to recreate the write calls is:
gawk -vFS='"' '/write.*/grid-coordinates.xy.asc/{print "printf ""$2"""}' ~/strace.18826.log >recreate.sh
The corrupted line is eg. line 161 of
/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/TEST/sim/CarpetIOASCII/newsep /grid-coordinates.xy.asc
which reads
1 4 3 4 1 0.1666666666660.505076272276105285714 etc
but should read
1 4 3 4 1 0.166666666666667 -0.0714285714285714
I can avoid the file corruption by flushing the output file after each line.
I am wondering if there is anything known about this or if there is a workaround that does not boil to first writing all data to a file system local to the compute node and copying to /oasis/scratch after the job is finished (how much local space would be available since I would also have to do so for eg checkpoint files and 3d hdf5 output).