#2023: CarpetInterp hangs with openmpi -----------------------------+---------------------------------------------- Reporter: anonymous | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Keywords: OpenMPI, Carpet | -----------------------------+---------------------------------------------- On three occasions with two different simulations (TOV, BNS) the code hung. In two cases, I could attach a debugger to the running process, and in both cases the backtrace looked like this: {{{ #0 0x00007f4ecfd5729a in __GI___pthread_mutex_lock (mutex=0xc40d730) at ../nptl/pthread_mutex_lock.c:79 #1 0x00007f4ec7224807 in ?? () from /usr/lib/openmpi/lib/openmpi/mca_btl_openib.so #2 0x00007f4ecdae734a in opal_progress () from /usr/lib/libmpi.so.1 #3 0x00007f4ecda2d3b4 in ompi_request_default_wait_all () from /usr/lib/libmpi.so.1 #4 0x00007f4ec61cb7b7 in ompi_coll_tuned_sendrecv_actual () from /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so #5 0x00007f4ec61d0df6 in ompi_coll_tuned_alltoallv_intra_pairwise () from /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so #6 0x00007f4ecda3a00f in PMPI_Alltoallv () from /usr/lib/libmpi.so.1 #7 0x00000000015458fe in CarpetInterp::Carpet_DriverInterpolate (cctkGH_=<optimized out>, N_dims=<optimized out>, local_interp_handle=3, param_table_handle=6421, coord_system_handle=0, N_interp_points=224, interp_coords_type_code=130, coords_list=<optimized out>, N_input_arrays=6, input_array_variable_indices=0x7fff38fbf640, N_output_arrays=24, output_array_type_codes=0x7fff38fbf660, output_arrays=0x7fff38fbfb00) at /home/wolfgang.kastaun/ET/Payne/Cactus/arrangements/Carpet/CarpetInterp/src/interp.cc:645 #8 0x000000000071ee4f in SymBase_SymmetryInterpolateFaces (cctkGH_=0xccb1be0, N_dims=<optimized out>, local_interp_handle=3, param_table_handle=6421, coord_system_handle=0, N_interp_points=224, interp_coords_type=130, interp_coords=0x7fff38fbe1a0, N_input_arrays=6, input_array_indices=0x7fff38fbf640, N_output_arrays=24, output_array_types=0x7fff38fbf660, output_arrays=0x7fff38fbfb00, faces=0) at /home/wolfgang.kastaun/ET/Payne/Cactus/arrangements/CactusBase/SymBase/src/Interpolation.c:381 }}}
I'm not sure if this is a problem with Carpet or with my OpenMPI installation. It only happens rarely, after one day or so. I'm using ET version Payne, gcc 4.9.2, openmpi 1.6.5 on an infiniband interconnect. The code was compiled with OpenMP support and run using 4 threads per process.
Looking at various variables in Carpet_DriverInterpolate with the debugger, I noticed that the vector tmp was reported with size -488221088, but this could also be mis-reported by gdb since the code was compiled with -O3.