#1547: issues with stampede -----------------------------+---------------------------------------------- Reporter: jchsma@… | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Other | Version: ET_2013_11 Resolution: | Keywords: -----------------------------+----------------------------------------------
Comment (by hinder):
I am also having similar problems on Stampede. I am using mvapich2. I posted (http://lists.einsteintoolkit.org/pipermail/users/2014-May/003580.html) a summary to the mailing list, and I include it here for reference.
I've had jobs die when checkpointing, and also mysteriously hanging for no apparent reason. These might be separate problems. The checkpointing issue occurred when I submitted several jobs and they all started checkpointing at the same time after 3 hours. The hang happened after a few hours of evolution, with GDB reporting
{{{ MPIDI_CH3I_MRAILI_Get_next_vbuf (vc_ptr=0x7fff00d9a8d8, vbuf_ptr=0x13) at src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:296 296 for (; i < mv2_MPIDI_CH3I_RDMA_Process.polling_group_size; ++i) }}}
Unfortunately I didn't ask for a backtrace. I'm using mvapich2. I've been in touch with support and they said the dying while checkpointing coincided with the filesystems being hit hard by my jobs, which makes sense, but they didn't see any problems in their logs, and they have no idea about the mysterious hang. I repeated the hanging job and it ran fine.