Hello Ian,
Do you have any more details?
I do, somehwere there must be emails either on this list or between myself and Milton Ruiz. The issue is a file system problem as one can actually use strace to track down the calls to the OS's seek and write functions and the result in files is obviously not what the read/write should generate (eg when one replays them).
I had the feeling that it was you who was fighting this. Would it be possible to add an autoconf or runtime test, so that users are at least made aware of the problem?
An autoconf test won't work since this only happens on the scratch file system and not the $HOME file system where one usually compiles and only happens for more than 1 MPI rank (even though only rank 0 ever touches that file).
Do you know of other systems that are affected? Do we have an ET ticket for this?
Gordon is also affected. The closest I can find to at ticket is the notice in the release coordination wiki page: https://docs.einsteintoolkit.org/et-docs/Release_coordination
There's a ticket with XSEDE about this, I guess I will ping it and also copy the extra information from it into an ET ticket.
Yours, Roland