#434: Keep track of masked-out volume in CarpetMask --------------------------+------------------------------------------------- Reporter: eschnett | Owner: eschnett Type: enhancement | Status: review Priority: minor | Milestone: ET_2013_05 Component: Carpet | Version: Resolution: | Keywords: --------------------------+-------------------------------------------------
Comment (by rhaas):
Does not seem to fix the issue though the number change slightly. I currently have
||= Code version =||= MPI processes =||= threads per process =||= Simulation domain volume =||= Additional excised volume =||= Reduction weight sum =|| || trunk || 6 || 6 || 431999.99795532227 || 0.002044677734375 || 431999.99794769287 || || trunk || 2 || 6 || 431999.99795532227 || 0.002044677734375 || 431999.99794769287 || || trunk || 8 || 6 || 431999.99795150757 || 0.002048492431640625 || 431999.99794769287 || || patched || 6 || 6 || 431999.99796295166 || 0.00203704833984375 || 431999.99794769287 || || patched || 1 || 1 || no difference || no difference || no difference || || patched || 36 || 1 || no difference || no difference || no difference || || patched || 6 || 1 || no difference || no difference || no difference ||
The expected total volume before exclusion (which is "Simulation domain valume"+"Additional excised volume") is 432000.
So it seems as if the error is due to some OpenMP issue since all runs using 1 thread per process find no difference whereas all runs that use 6 threads per process do find a difference.
Please note that *all* runs (with or without the patch, with any number of processes and threads) abort shortly afterwards with a SEGFAULT exception: {{{ [zwicky163:02171] [ 0] /lib64/libpthread.so.0 [0x2b1c3e286be0] [zwicky163:02171] [ 1] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(opal_memory_ptmalloc2_int_malloc+0x1fa) [0x2c4bf3a] [zwicky163:02171] [ 2] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim [0x2c4ab58] [zwicky163:02171] [ 3] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x2b1c3ff081dd] [zwicky163:02171] [ 4] /usr/lib64/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x21) [0x2b1c3fee6861] [zwicky163:02171] [ 5] /usr/lib64/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x2b) [0x2b1c3fee723b] [zwicky163:02171] [ 6] /usr/lib64/libstdc++.so.6(_ZNSs7reserveEm+0x45) [0x2b1c3fee7b45] [zwicky163:02171] [ 7] /usr/lib64/libstdc++.so.6(_ZNSt15basic_stringbufIcSt11char_traitsIcESaIcEE8overflowEi+0xd5) [0x2b1c3fee10c5] [zwicky163:02171] [ 8] /usr/lib64/libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x7d) [0x2b1c3fee5d1d] [zwicky163:02171] [ 9] /usr/lib64/libstdc++.so.6(_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc+0x111) [0x2b1c3fedb311] [zwicky163:02171] [10] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN6Carpet15reflevel_setterC1EPK4_cGHi+0x1f8) [0x183f608] [zwicky163:02171] [11] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN12CarpetReduce9ReduceGVsEPK4_cGHiiiPviPKiPKNS_9reductionEi+0x45e) [0x17bf3fe] [zwicky163:02171] [12] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN12CarpetReduce11average_GVsEPK4_cGHiiiPviPKi+0x2d) [0x17c201d] [zwicky163:02171] [13] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTK_Reduce+0x148) [0x7d9fc8] [zwicky163:02171] [14] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim [0x77f430] [zwicky163:02171] [15] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim [0x77e48a] [zwicky163:02171] [16] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTKi_TriggerAction+0x66) [0x5acd56] [zwicky163:02171] [17] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim [0x5a947b] [zwicky163:02171] [18] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTKi_DoScheduleTraverse+0x208) [0x5ac0e8] [zwicky163:02171] [19] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTK_ScheduleTraverse+0x199) [0x5a5119] [zwicky163:02171] [20] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim [0x17db30d] [zwicky163:02171] [21] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN6Carpet10InitialiseEP12tFleshConfig+0x2e7) [0x17de647] [zwicky163:02171] [22] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(main+0x99) [0x59a3b9] [zwicky163:02171] [23] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b1c4057b994] [zwicky163:02171] [24] /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim [0x59a0b9] [zwicky163:02171] *** End of error message *** }}}
which comes from Carpet/Carpet/src/modes.cc:1359 which is a call to enter_level_mode.
If either one of these is reproducible than I think we must bump this issue to critical and fix it before the release.
These happened with a clean (according to a diff against a fresh checked out copy) -realcleaned configuration. I am trying this on lonestar now.