#2073: IO corruption on SDSC oasis file systems -------------------------------+-------------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Other | Version: development version Keywords: Comet Gordon SDSC | -------------------------------+-------------------------------------------- I am experiencing file corruption in ASCII output produced by the Cactus code on Comet's (and Gordon's as far as I remember) scratch file systems. This manifests as lines of output being mashed together in the output file.

I have added strace calls to my job script to capture all arguments to the OS's write() function and re-created the write() calls based on this. Those write calls, when replayed on a login node, do produce a correct (no mashed lines) file.

All output to the file in question was from rank 0 only even though the code used MPI and ran on two MPI ranks.

The same code and number of MPI ranks produces a correct output file when run on the $HOME file system.

Thus it seems to me as if there may be an issue with the file system. I can try and reduce the test case to a more minimal example (right now it is a full simulation even though it runs only for <1minute) .

You can find the job script (for account, SLURM options etc) here:

/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/SIMFACTORY/SubmitScript

the script that launches the MPI executable here:

/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/SIMFACTORY/RunScript

the strace output here:

/home/rhaas/strace/strace.1882[67].log

and the awk script to recreate the write calls is:

gawk -vFS='"' '/write.*/grid-coordinates.xy.asc/{print "printf ""$2"""}' ~/strace.18826.log >recreate.sh

The corrupted line is eg. line 161 of

/oasis/scratch/comet/rhaas/temp_project/simulations/OSTREAM_2_12/output-0000/TEST/sim/CarpetIOASCII/newsep /grid-coordinates.xy.asc

which reads

1 4 3 4 1 0.1666666666660.505076272276105285714 etc

but should read

1 4 3 4 1 0.166666666666667 -0.0714285714285714

I can avoid the file corruption by flushing the output file after each line.

I am wondering if there is anything known about this or if there is a workaround that does not boil to first writing all data to a file system local to the compute node and copying to /oasis/scratch after the job is finished (how much local space would be available since I would also have to do so for eg checkpoint files and 3d hdf5 output).