On 07/28/2014 06:30 AM, Ian Hinder wrote:
On 21 Jul 2014, at 21:52, Frank Loeffler knarf@cct.lsu.edu wrote:
- Josef: has problems on stampede, trouble with MPI and checkpointing suggestion: attaching debugger to see where it hangs. Something might have changed in recent ET releases that makes issues worse now, but it is not sure whom to blame. We might just now trigger a problem with the stampede system.
If it is a hang, then I would look at the IOScalar reduction output. I had a hang, and attached GDB, and got the following backtrace on 2-May-2014:
I ran into the issue that short runs never stalled. I tried a run that checkpointed every few iterations and it worked fine. Long runs (> 4 hours) stall, but I ran into the issue of catching the stall before the system killed the run (not so easy when the queue time was up to 12 hours). I found that increasing the number of nodes from 16 to 20 seem to "fix" the problem. The issue there is that 20 nodes is not ideal due to scaling (the runs use about 12GB per node).
MPIDI_CH3I_MRAILI_Cq_poll (vbuf_handle=0x79c8c50, vc_req=0x1, receiving=-2076533752, is_blocking=1) at src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:920 920 if (type == T_CHANNEL_NO_ARRIVE) { Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 gsl-intel13-1.15-1.x86_64 libgcc-4.4.7-4.el6.x86_64 libibmad-1.3.8-1.x86_64 libibumad-1.3.7-1.x86_64 libibverbs-1.1.4-1.24.gb89d4d7.x86_64 libmlx4-1.0.1-1.20.g6771d22.x86_64 libmthca-1.0.5-0.1.gbe5eef3.x86_64 librdmacm-1.0.15-1.x86_64 libstdc++-4.4.7-4.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 MPIDI_CH3I_MRAILI_Cq_poll (vbuf_handle=0x79c8c50, vc_req=0x1, receiving=-2076533752, is_blocking=1) at src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:920 #1 0x00002b9c83f01539 in MPIDI_CH3I_read_progress (vc_pptr=0x79c8c50, v_ptr=0x1, rdmafp_found=0x2b9c843a9c08, is_blocking=1) at src/mpid/ch3/channels/mrail/src/rdma/ch3_read_progress.c:148 #2 0x00002b9c83eff011 in cm_handle_pending_send (is_blocking=127700048, state=0x1) at src/mpid/ch3/channels/mrail/src/rdma/ch3_progress.c:249 #3 MPIDI_CH3I_Progress (is_blocking=127700048, state=0x1) at src/mpid/ch3/channels/mrail/src/rdma/ch3_progress.c:244 #4 0x00002b9c84035e9a in PMPI_Waitany (count=127700048, array_of_requests=0x1, index=0x2b9c843a9c08, status=0x1) at src/mpi/pt2pt/waitany.c:198 #5 0x00002b9c83fef027 in MPIR_Reduce_knomial_MV2 (sendbuf=0x79c8c50, recvbuf=0x1, count=1, datatype=1275069445, op=127410992, root=-800453280, comm_ptr=0x7b87320, errflag=0x7fffd04a1178) at src/mpi/coll/reduce_osu.c:985 #6 0x00002b9c83fef8ef in MPIR_Reduce_two_level_helper_MV2 (sendbuf=0x79c8c50, recvbuf=0x1, count=-2076533752, datatype=1, op=127410992, root=-800453280, comm_ptr=0x7b86d68, errflag=0x2b9c83ff041b) at src/mpi/coll/reduce_osu.c:1267 #7 0x00002b9c83ff041b in MPIR_Reduce_MV2 (sendbuf=0x79c8c50, recvbuf=0x1, count=-2076533752, datatype=1, op=127410992, root=-800453280, comm_ptr=0x0, errflag=0x2b9c83f8a206) at src/mpi/coll/reduce_osu.c:1484 #8 0x00002b9c83f8a206 in MPIR_Reduce_impl (sendbuf=0x79c8c50, recvbuf=0x1, count=-2076533752, datatype=1, op=127410992, root=-800453280, comm_ptr=0x7fffd04a4ea0, errflag=0x2b9c83f8c01e) at src/mpi/coll/reduce.c:1029 #9 0x00002b9c83f8c01e in PMPI_Reduce (sendbuf=0x79c8c50, recvbuf=0x1, count=-2076533752, datatype=1, op=127410992, root=-800453280, comm=-800436576) at src/mpi/coll/reduce.c:1216 #10 0x0000000000b76bf7 in CarpetReduce::Finalise (cgh=0x79c8c50, proc=1, num_outvals=-2076533752, outvals=0x1, outtype=127410992, myoutvals=0x7fffd04a0d60, mycounts=0x1, red=0xb79231) at /work/00915/hinder/Cactus/arrangements/Carpet/CarpetReduce/src/reduce.cc:954 #11 0x0000000000b79231 in CarpetReduce::ReduceGVs (cgh=0x79c8c50, proc=1, num_outvals=-2076533752, outtype=1, outvals=0x7982330, num_invars=-800453280, invars=0x9ae12b0, red=0x7fffd04a16b0, igrid=0) at /work/00915/hinder/Cactus/arrangements/Carpet/CarpetReduce/src/reduce.cc:1584 #12 0x0000000000b7ad8d in CarpetReduce::maximum_GVs (cgh=0x79c8c50, proc=1, num_outvals=-2076533752, outtype=1, outvals=0x7982330, num_invars=-800453280, invars=0x200000000) at /work/00915/hinder/Cactus/arrangements/Carpet/CarpetReduce/src/reduce.cc:1622 #13 0x00000000006eded0 in CCTK_Reduce (GH=0x79c8c50, proc=1, operation_handle=-2076533752, num_out_vals=1, type_out_vals=127410992, out_vals=0x7fffd04a0d60, num_in_fields=102) at /work/00915/hinder/Cactus/src/comm/Reduction.c:429 #14 0x00000000006d2f5a in CarpetIOScalar::OutputVarAs (cctkGH=0x79c8c50, varname=0x1 <Address 0x1 out of bounds>, alias=0x2b9c843a9c08 "\001", out_reductions=0x1 <Address 0x1 out of bounds>) at /work/00915/hinder/Cactus/arrangements/Carpet/CarpetIOScalar/src/ioscalar.cc:461 #15 0x00000000006d1f0c in CarpetIOScalar::TriggerOutput (cctkGH=0x79c8c50, vindex=1) at /work/00915/hinder/Cactus/arrangements/Carpet/CarpetIOScalar/src/ioscalar.cc:731 #16 0x00000000006d1cc3 in CarpetIOScalar::OutputGH (cctkGH=0x79c8c50) at /work/00915/hinder/Cactus/arrangements/Carpet/CarpetIOScalar/src/ioscalar.cc:170 #17 0x0000000000bd2bee in Carpet::OutputGH (cctkGH=0x79c8c50) at /work/00915/hinder/Cactus/arrangements/Carpet/Carpet/src/OutputGH.cc:56 #18 0x0000000000bc81c2 in Carpet::CallAnalysis (cctkGH=0x79c8c50) at /work/00915/hinder/Cactus/arrangements/Carpet/Carpet/src/Evolve.cc:767 #19 0x0000000000bc70e3 in Carpet::Evolve (fc=0x79c8c50) at /work/00915/hinder/Cactus/arrangements/Carpet/Carpet/src/Evolve.cc:80 #20 0x0000000000575bc1 in main (argc=4, argv=0x7fffd04a5a28) at /work/00915/hinder/Cactus/src/main/flesh.cc:84
I would have been using Carpet 2ede3e9156b91bb5061802cdf9e50dca2bb2114d (01-Apr-2014) plus some local commits, so somewhere on the trunk between ET_2013_11 (Noether) and ET_2014_15 (Wheeler).
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users