[ET Trac] [Einstein Toolkit] #970: CarpetLib::barriers fails with multipatch

[Einstein Toolkit]

#970: CarpetLib::barriers fails with multipatch
--------------------+-------------------------------------------------------
 Reporter:  rhaas   |       Owner:  eschnett
     Type:  defect  |      Status:  new     
 Priority:  minor   |   Milestone:          
Component:  Carpet  |     Version:          
 Keywords:          |  
--------------------+-------------------------------------------------------
 This happens during the initial storage allocation where there are
 mismatching barriers in dh::add and gdata::gdata. The underlying reason
 seems to be that Carpet/StorageCrease has a loop (schematically) around
 line 93 of Storage.cc.

 {{
 for(m=0;m<maps;++m)
 new gf<T> // (which calls dh::add)

 arrdata.AT(group).AT(m).data.AT(var)->set_timelevels // which eventually
 call gdata::gdata
 }}

 this causes the a barrier error when on process owns a component on map 0
 but another does only onwn a component on map 1, since in this case the
 first one will encounter the barriers as:

 dhd::add (map 0)
 gdata::gdata (component on map 0)
 dh::add (map 1)

 while the other process sees:

 dhd::add (map 0)
 gdata::gdata (component on map 0)
 dh::add (map 1)

 The actual error is then (where there are some extra printf() lines that I
 added):
 {{{
 INFO (Carpet): [tl=0] Starting initialisation
 INFO (Carpet): [tl=0] GroupStorageIncrease
 INFO (Carpet): [tl=0]   ADMBASE::SHIFT_STATE: increase to 1
 dh::add added varindex 0: shift_state
 CHECKPOINT: processor 16, file
 /work/00945/rhaas/Zelmani/arrangements/Carpet/CarpetLib/src/dh.cc, line
 2176
 Adding varindex 0: shift_state
 INFO (Carpet): [tl=0]   ADMBASE::DTLAPSE_STATE: increase to 1
 dh::add added varindex 1: dtlapse_state
 CHECKPOINT: processor 16, file
 /work/00945/rhaas/Zelmani/arrangements/Carpet/CarpetLib/src/dh.cc, line
 2176
 Adding varindex 1: dtlapse_state
 INFO (Carpet): [tl=0]   ADMBASE::DTSHIFT_STATE: increase to 1
 dh::add added varindex 2: dtshift_state
 CHECKPOINT: processor 16, file
 /work/00945/rhaas/Zelmani/arrangements/Carpet/CarpetLib/src/dh.cc, line
 2176
 Adding varindex 2: dtshift_state
 INFO (Carpet): [tl=0]   ADMBASE::LAPSE: increase to 1
 dh::add added varindex 15: alp
 CHECKPOINT: processor 16, file
 /work/00945/rhaas/Zelmani/arrangements/Carpet/CarpetLib/src/dh.cc, line
 2176
 dh::add added varindex 15: alp
 CHECKPOINT: processor 16, file
 /work/00945/rhaas/Zelmani/arrangements/Carpet/CarpetLib/src/dh.cc, line
 2176
 WARNING level 0 in thorn CarpetLib processor 16 host
 c305-212.ls4.tacc.utexas.edu
   (line 251 of
 /work/00945/rhaas/Zelmani/arrangements/Carpet/CarpetLib/src/dist.cc):
   -> Wrong id for Barrier "CarpetLib::dist::checkpoint": expected
 506880075d, found 783988953d
 }}}

 This like something that is rather hard to fix generally for little
 benefit (ie. it affects only debugging runs with multipatch). Should this
 even be reported (if only so that there is official notice that this is
 known behaviour)? Should the fix be just a warning if Carpet encounters
 this situation?

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/970>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit