[ET Trac] [Einstein Toolkit] #434: Keep track of masked-out volume in CarpetMask

[Einstein Toolkit]

#434: Keep track of masked-out volume in CarpetMask
--------------------------+-------------------------------------------------
  Reporter:  eschnett     |       Owner:  eschnett  
      Type:  enhancement  |      Status:  review    
  Priority:  minor        |   Milestone:  ET_2013_05
 Component:  Carpet       |     Version:            
Resolution:               |    Keywords:            
--------------------------+-------------------------------------------------

Comment (by rhaas):

 Does not seem to fix the issue though the number change slightly. I
 currently have

 ||= Code version =||= MPI processes =||= threads per process =||=
 Simulation domain volume =||= Additional excised volume =||= Reduction
 weight sum =||
 || trunk   || 6 || 6 || 431999.99795532227 || 0.002044677734375 ||
 431999.99794769287 ||
 || trunk   || 2 || 6 || 431999.99795532227 || 0.002044677734375 ||
 431999.99794769287 ||
 || trunk   || 8 || 6 || 431999.99795150757 || 0.002048492431640625 ||
 431999.99794769287 ||
 || patched || 6 || 6 || 431999.99796295166 || 0.00203704833984375 ||
 431999.99794769287 ||
 || patched || 1 || 1 || no difference      || no difference     || no
 difference      ||
 || patched || 36 || 1 || no difference      || no difference     || no
 difference      ||
 || patched || 6 || 1 || no difference      || no difference     || no
 difference      ||

 The expected total volume before exclusion (which is "Simulation domain
 valume"+"Additional excised volume") is 432000.

 So it seems as if the error is due to some OpenMP issue since all runs
 using 1 thread per process find no difference whereas all runs that use 6
 threads per process do find a difference.

 Please note that *all* runs (with or without the patch, with any number of
 processes and threads) abort shortly afterwards with a SEGFAULT exception:
 {{{
 [zwicky163:02171] [ 0] /lib64/libpthread.so.0 [0x2b1c3e286be0]
 [zwicky163:02171] [ 1]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(opal_memory_ptmalloc2_int_malloc+0x1fa)
 [0x2c4bf3a]
 [zwicky163:02171] [ 2]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim
 [0x2c4ab58]
 [zwicky163:02171] [ 3] /usr/lib64/libstdc++.so.6(_Znwm+0x1d)
 [0x2b1c3ff081dd]
 [zwicky163:02171] [ 4]
 /usr/lib64/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x21)
 [0x2b1c3fee6861]
 [zwicky163:02171] [ 5]
 /usr/lib64/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x2b)
 [0x2b1c3fee723b]
 [zwicky163:02171] [ 6] /usr/lib64/libstdc++.so.6(_ZNSs7reserveEm+0x45)
 [0x2b1c3fee7b45]
 [zwicky163:02171] [ 7]
 /usr/lib64/libstdc++.so.6(_ZNSt15basic_stringbufIcSt11char_traitsIcESaIcEE8overflowEi+0xd5)
 [0x2b1c3fee10c5]
 [zwicky163:02171] [ 8]
 /usr/lib64/libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x7d)
 [0x2b1c3fee5d1d]
 [zwicky163:02171] [ 9]
 /usr/lib64/libstdc++.so.6(_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc+0x111)
 [0x2b1c3fedb311]
 [zwicky163:02171] [10]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN6Carpet15reflevel_setterC1EPK4_cGHi+0x1f8)
 [0x183f608]
 [zwicky163:02171] [11]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN12CarpetReduce9ReduceGVsEPK4_cGHiiiPviPKiPKNS_9reductionEi+0x45e)
 [0x17bf3fe]
 [zwicky163:02171] [12]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN12CarpetReduce11average_GVsEPK4_cGHiiiPviPKi+0x2d)
 [0x17c201d]
 [zwicky163:02171] [13]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTK_Reduce+0x148)
 [0x7d9fc8]
 [zwicky163:02171] [14]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim
 [0x77f430]
 [zwicky163:02171] [15]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim
 [0x77e48a]
 [zwicky163:02171] [16]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTKi_TriggerAction+0x66)
 [0x5acd56]
 [zwicky163:02171] [17]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim
 [0x5a947b]
 [zwicky163:02171] [18]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTKi_DoScheduleTraverse+0x208)
 [0x5ac0e8]
 [zwicky163:02171] [19]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(CCTK_ScheduleTraverse+0x199)
 [0x5a5119]
 [zwicky163:02171] [20]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim
 [0x17db30d]
 [zwicky163:02171] [21]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(_ZN6Carpet10InitialiseEP12tFleshConfig+0x2e7)
 [0x17de647]
 [zwicky163:02171] [22]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim(main+0x99)
 [0x59a3b9]
 [zwicky163:02171] [23] /lib64/libc.so.6(__libc_start_main+0xf4)
 [0x2b1c4057b994]
 [zwicky163:02171] [24]
 /panfs/ds06/sxs/rhaas/simulations/qc0-mclachlanOersted_fixed/SIMFACTORY/exe/cactus_sim
 [0x59a0b9]
 [zwicky163:02171] *** End of error message ***
 }}}

 which comes from Carpet/Carpet/src/modes.cc:1359 which is a call to
 enter_level_mode.

 If either one of these is reproducible than I think we must bump this
 issue to critical and fix it before the release.

 These happened with a clean (according to a diff against a fresh checked
 out copy) -realcleaned configuration. I am trying this on lonestar now.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/434#comment:18>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit