Hi Roland,
In the release notes for the current release, it is written
http://einsteintoolkit.org/about/releases/ET_2017_06_announcement.html:
-(#) On Comet, using more than one MPI process might produce corrupt output files. This is a known file system bug on this machine.
https://docs.einsteintoolkit.org/et-docs/Release_coordination:
• comet • failures on multiple MPI ranks are almost certainly due to the file system bug that corrupts ASCII output files. Nothing we can do about it at this point (there is a ticket by Milton Ruiz with SDSC) short of adding various strange workarounds to every output routine (and it may still fail for checkpoint and recovery as is happening to the SpEC code right now)
Do you have any more details? I had the feeling that it was you who was fighting this. Would it be possible to add an autoconf or runtime test, so that users are at least made aware of the problem? Do you know of other systems that are affected? Do we have an ET ticket for this?
Hello Ian,
Do you have any more details?
I do, somehwere there must be emails either on this list or between myself and Milton Ruiz. The issue is a file system problem as one can actually use strace to track down the calls to the OS's seek and write functions and the result in files is obviously not what the read/write should generate (eg when one replays them).
I had the feeling that it was you who was fighting this. Would it be possible to add an autoconf or runtime test, so that users are at least made aware of the problem?
An autoconf test won't work since this only happens on the scratch file system and not the $HOME file system where one usually compiles and only happens for more than 1 MPI rank (even though only rank 0 ever touches that file).
Do you know of other systems that are affected? Do we have an ET ticket for this?
Gordon is also affected. The closest I can find to at ticket is the notice in the release coordination wiki page: https://docs.einsteintoolkit.org/et-docs/Release_coordination
There's a ticket with XSEDE about this, I guess I will ping it and also copy the extra information from it into an ET ticket.
Yours, Roland
users@lists.einsteintoolkit.org
-
Ian Hinder -
Roland Haas