[ET Trac] [Einstein Toolkit] #626: Recovery fails in AHFinderDirect RecoverML with out-of-bounds assertion in CarpetLib

Wed May 23 13:53:46 CDT 2012

#626: Recovery fails in AHFinderDirect RecoverML with out-of-bounds assertion in
CarpetLib
-----------------------+----------------------------------------------------
  Reporter:  rhaas     |       Owner:  eschnett
      Type:  defect    |      Status:  new     
  Priority:  critical  |   Milestone:          
 Component:  Carpet    |     Version:          
Resolution:            |    Keywords:          
-----------------------+----------------------------------------------------

Comment (by rhaas):

 Erik: for the ML tests your method works for me. It would not work had I
 set eg. ADMBase::metric_timelevels = 3 initially since that parameter is
 not steerable. Ie. adding
 {{{
 ADMBase::metric_timelevels = 3
 }}}
 to the checkpointML.par from the test cannot be fixed since the parameter
 is not steerable (shouldn't Cactus also complain if I try to steer a
 parameter upon recovery the is not steerable?). The backtrace is:

 {{{
  #0  0x00007ffff4305475 in *__GI_raise (sig=<optimized out>) at
 ../nptl/sysdeps/unix/sysv/linux/raise.c:64
 #1  0x00007ffff43086f0 in *__GI_abort () at abort.c:92
 #2  0x00007ffff42fe621 in *__GI___assert_fail (assertion=0x37cb2e5 "tl>=0
 and tl<timelevels", file=<optimized out>, line=79,
     function=0x3806540 "double th::get_time(int, int, int) const") at
 assert.c:81
 #3  0x0000000001664829 in th::get_time (this=<optimized out>,
 ml=<optimized out>, rl=<optimized out>, tl=<optimized out>)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetLib/src/th.hh:79
 #4  0x000000000185c96a in transfer_from_all (slabinfo=0x0,
 use_old_storage=false, ml2=0, rl2=0, tl2=2,
 sendrecvs=&dh::fast_dboxes::fast_sync_sendrecv, ml1=0,
     rl1=0, tl1=2, state=..., this=0x79638f0, flip_send_recv=<optimized
 out>) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetLib/src/ggf.hh:184
 #5  ggf::sync_all (this=0x79638f0, state=..., tl=2, rl=0, ml=0) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetLib/src/ggf.cc:344
 #6  0x000000000166324b in CarpetIOHDF5::Recover (cctkGH=0x78d3360,
 basefilename=<optimized out>, called_from=3)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetIOHDF5/src/Input.cc:762
 #7  0x0000000003214381 in IOUtil_RecoverFromFile (GH=0x78d3360,
 basefilename=0x0, called_from=3)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/CactusBase/IOUtil/src/CheckpointRecovery.c:335
 #8  0x0000000003214ac4 in IOUtil_RecoverGH (GH=<optimized out>)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/CactusBase/IOUtil/src/CheckpointRecovery.c:367
 #9  0x000000000056b3c5 in CCTK_CallFunction (function=0x3214aa0,
 fdata=0x78b74d8, data=0x78d3360)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:311
 #10 0x00000000017ace5f in Carpet::CallScheduledFunction
 (time_and_mode=<optimized out>, function=0x3214aa0, attribute=0x78b74d8,
 data=0x78d3360, user_timer=...)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/CallFunction.cc:367
 #11 0x00000000017af5a8 in Carpet::CallFunction (function=0x3214aa0,
 attribute=0x78b74d8, data=0x78d3360)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/CallFunction.cc:260
 #12 0x000000000056b101 in CCTKi_ScheduleCallFunction (function=0x3214aa0,
 attribute=0x78b74c0, data=0x7fffffffd340)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:3027
 #13 0x000000000057513b in ScheduleTraverseFunction (data=<optimized out>,
 function_process=<optimized out>, if_check=<optimized out>,
 while_check=<optimized out>,
     item_exit=<optimized out>, item_entry=<optimized out>, ifs=<optimized
 out>, n_ifs=<optimized out>, whiles=0x0, n_whiles=<optimized out>,
 attributes=0x78b74c0,
     function=0x3214aa0) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/schedule/ScheduleTraverse.c:584
 #14 ScheduleTraverseGroup (schedule_groups=0x7892280, group=<optimized
 out>, attributes=0x0, n_whiles=0, whiles=0x0, n_ifs=<optimized out>,
 ifs=0x0,
     item_entry=0x56abd0 <CCTKi_ScheduleCallEntry>, item_exit=0x56b750
 <CCTKi_ScheduleCallExit>, while_check=0x56a880 <CCTKi_ScheduleCallWhile>,
     if_check=0x56a530 <CCTKi_ScheduleCallIf>, function_process=0x56b0a0
 <CCTKi_ScheduleCallFunction>, data=0x7fffffffd340)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/schedule/ScheduleTraverse.c:368
 #15 0x00000000005752ff in CCTKi_DoScheduleTraverse (group_name=0x37267a4
 "CCTK_RECOVER_VARIABLES", item_entry=0x56abd0 <CCTKi_ScheduleCallEntry>,
     item_exit=0x56b750 <CCTKi_ScheduleCallExit>, while_check=0x56a880
 <CCTKi_ScheduleCallWhile>, if_check=0x56a530 <CCTKi_ScheduleCallIf>,
     function_process=<optimized out>, data=0x7fffffffd340) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/schedule/ScheduleTraverse.c:158
 #16 0x00000000005707db in ScheduleTraverse (CallFunction=0x17addc0
 <Carpet::CallFunction(void*, cFunctionData*, void*)>, GH=0x78d3360,
     where=0x37267a4 "CCTK_RECOVER_VARIABLES") at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:1360
 #17 CCTK_ScheduleTraverse (where=0x37267a4 "CCTK_RECOVER_VARIABLES",
 GH=0x78d3360, CallFunction=0x17addc0 <Carpet::CallFunction(void*,
 cFunctionData*, void*)>)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:891
 #18 0x0000000001738d11 in Carpet::ScheduleTraverse (name=0x37267a4
 "CCTK_RECOVER_VARIABLES", cctkGH=0x78d3360, where=<optimized out>)
     at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/Initialise.cc:1277
 #19 0x00000000017396d9 in CallRecoverVariables (cctkGH=0x78d3360) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/Initialise.cc:241
 #20 Carpet::Initialise (fc=0x7fffffffd7e0) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/Initialise.cc:115
 #21 0x00000000005633a3 in main (argc=2, argv=0x7fffffffd8e8) at
 /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/flesh.cc:80
 }}}

 this is revision 1566 of AHFinderDirect and Carpet hash 3555:12ee1105096a
 attached are the parameter files (ending in -admbase) and stdout as well
 as stderr. This is with one MPI process and one thread. On my workstation
 at Caltech.

 The inability to solve via a parameter setting is thus mostly related to
 some parameters not being steerable upon recovery even though they could
 be changed at least in some ways without problem.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/626#comment:16>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit