#2023: CarpetInterp hangs with openmpi ------------------------+--------------------------------------------------- Reporter: anonymous | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: OpenMPI, Carpet ------------------------+---------------------------------------------------
Comment (by anonymous):
Replying to [comment:1 hinder]:
This looks very similar to what we encountered with OmniPath on Minerva,
which held up the acceptance for a few months. In that case, it was a bug in the OmniPath driver, which Intel eventually found and fixed, using our reproducible example. This was with SpEC, not Cactus. It looks the same on surface (a
Is that reproducible example a full SpEC run or a small example code I could test as well? Did you use hyperthreading and/or OpenMP?
hang in MPI_Alltoallv), but you are using OpenMPI and infiniband, whereas we were using Intel MPI and OmniPath, so unfortunately this doesn't seem to be the same problem. You could also try compiling without optimisation to see if that gives better backtraces.
Is your problem reproducible?
Not really. I did several short benchmarks (<20 min) with the same executable, and it happened to 2 out of 18. It also happened in one longer run after around 30 hours.
What cluster is this?
The new cluster "holodeck" for NR at AEI Hanover. It's a 640 core Intel Xeon system.
Have you tried different OpenMPI versions, or a different MPI
implementation?
Not yet, we need to install them first..