Hi,
I've noticed that I get a number of jobs failing on stampede with the following error:
[59:c453-703][../../dapl_poll_rc.c:1360] Intel MPI fatal error: ofa-v2-mlx4_0-1 DTO operation posted for [1:c411-103] completed with error. status=0x8. cookie=0x0 Assertion failed in file ../../dapl_poll_rc.c at line 1362: 0 internal ABORT - process 59
It appears to be intermittent; if I restart from the last checkpoint, the simulation gets a bit further. Switching to MVAPICH appears to improve things (time will tell) but it seems to be significantly slower than Intel MPI. I remember also having problems on SuperMUC with Intel MPI which were solved by switching to IBM MPI there.
Has anyone else seen these problems, and if so, do you have a workaround? I'm using the standard simfactory optionlist with 512 cores and num-threads = 8.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello Ian,
Has anyone else seen these problems, and if so, do you have a workaround? I'm using the standard simfactory optionlist with 512 cores and num-threads = 8.
I have seen these problems with SpEC as well (num-threads=1, about 500 cores). Eg:
[170:c425-501] unexpected disconnect completion event from [26:c407-203] Assertion failed in file ../../dapl_conn_rc.c at line 1174: 0 internal ABORT - process 170
I have no workaround unfortunately.
With SpEC I also saw ever increasing memory consumption when using mvapich (though as far as I could tell from mallinfo there was no sign of memory fragmenting the same way we had it in Carpet recently I think).
Yours, Roland
- -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
On 19 Sep 2013, at 17:38, Roland Haas roland.haas@physics.gatech.edu wrote:
Signed PGP part Hello Ian,
Has anyone else seen these problems, and if so, do you have a workaround? I'm using the standard simfactory optionlist with 512 cores and num-threads = 8.
I have seen these problems with SpEC as well (num-threads=1, about 500 cores). Eg:
[170:c425-501] unexpected disconnect completion event from [26:c407-203] Assertion failed in file ../../dapl_conn_rc.c at line 1174: 0 internal ABORT - process 170
I have no workaround unfortunately.
With SpEC I also saw ever increasing memory consumption when using mvapich (though as far as I could tell from mallinfo there was no sign of memory fragmenting the same way we had it in Carpet recently I think).
Thanks Roland. I think I was also seeing increased memory consumption. It's a shame because I think Intel MPI was maybe (from memory) 10% faster in overall run speed.
users@lists.einsteintoolkit.org
-
Ian Hinder -
Roland Haas