#626: Recovery fails in AHFinderDirect RecoverML with out-of-bounds assertion in CarpetLib -----------------------+---------------------------------------------------- Reporter: rhaas | Owner: eschnett Type: defect | Status: new Priority: critical | Milestone: Component: Carpet | Version: Resolution: | Keywords: -----------------------+----------------------------------------------------
Comment (by rhaas):
Erik: for the ML tests your method works for me. It would not work had I set eg. ADMBase::metric_timelevels = 3 initially since that parameter is not steerable. Ie. adding {{{ ADMBase::metric_timelevels = 3 }}} to the checkpointML.par from the test cannot be fixed since the parameter is not steerable (shouldn't Cactus also complain if I try to steer a parameter upon recovery the is not steerable?). The backtrace is:
{{{ #0 0x00007ffff4305475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007ffff43086f0 in *__GI_abort () at abort.c:92 #2 0x00007ffff42fe621 in *__GI___assert_fail (assertion=0x37cb2e5 "tl>=0 and tl<timelevels", file=<optimized out>, line=79, function=0x3806540 "double th::get_time(int, int, int) const") at assert.c:81 #3 0x0000000001664829 in th::get_time (this=<optimized out>, ml=<optimized out>, rl=<optimized out>, tl=<optimized out>) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetLib/src/th.hh:79 #4 0x000000000185c96a in transfer_from_all (slabinfo=0x0, use_old_storage=false, ml2=0, rl2=0, tl2=2, sendrecvs=&dh::fast_dboxes::fast_sync_sendrecv, ml1=0, rl1=0, tl1=2, state=..., this=0x79638f0, flip_send_recv=<optimized out>) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetLib/src/ggf.hh:184 #5 ggf::sync_all (this=0x79638f0, state=..., tl=2, rl=0, ml=0) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetLib/src/ggf.cc:344 #6 0x000000000166324b in CarpetIOHDF5::Recover (cctkGH=0x78d3360, basefilename=<optimized out>, called_from=3) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/CarpetIOHDF5/src/Input.cc:762 #7 0x0000000003214381 in IOUtil_RecoverFromFile (GH=0x78d3360, basefilename=0x0, called_from=3) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/CactusBase/IOUtil/src/CheckpointRecovery.c:335 #8 0x0000000003214ac4 in IOUtil_RecoverGH (GH=<optimized out>) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/CactusBase/IOUtil/src/CheckpointRecovery.c:367 #9 0x000000000056b3c5 in CCTK_CallFunction (function=0x3214aa0, fdata=0x78b74d8, data=0x78d3360) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:311 #10 0x00000000017ace5f in Carpet::CallScheduledFunction (time_and_mode=<optimized out>, function=0x3214aa0, attribute=0x78b74d8, data=0x78d3360, user_timer=...) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/CallFunction.cc:367 #11 0x00000000017af5a8 in Carpet::CallFunction (function=0x3214aa0, attribute=0x78b74d8, data=0x78d3360) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/CallFunction.cc:260 #12 0x000000000056b101 in CCTKi_ScheduleCallFunction (function=0x3214aa0, attribute=0x78b74c0, data=0x7fffffffd340) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:3027 #13 0x000000000057513b in ScheduleTraverseFunction (data=<optimized out>, function_process=<optimized out>, if_check=<optimized out>, while_check=<optimized out>, item_exit=<optimized out>, item_entry=<optimized out>, ifs=<optimized out>, n_ifs=<optimized out>, whiles=0x0, n_whiles=<optimized out>, attributes=0x78b74c0, function=0x3214aa0) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/schedule/ScheduleTraverse.c:584 #14 ScheduleTraverseGroup (schedule_groups=0x7892280, group=<optimized out>, attributes=0x0, n_whiles=0, whiles=0x0, n_ifs=<optimized out>, ifs=0x0, item_entry=0x56abd0 <CCTKi_ScheduleCallEntry>, item_exit=0x56b750 <CCTKi_ScheduleCallExit>, while_check=0x56a880 <CCTKi_ScheduleCallWhile>, if_check=0x56a530 <CCTKi_ScheduleCallIf>, function_process=0x56b0a0 <CCTKi_ScheduleCallFunction>, data=0x7fffffffd340) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/schedule/ScheduleTraverse.c:368 #15 0x00000000005752ff in CCTKi_DoScheduleTraverse (group_name=0x37267a4 "CCTK_RECOVER_VARIABLES", item_entry=0x56abd0 <CCTKi_ScheduleCallEntry>, item_exit=0x56b750 <CCTKi_ScheduleCallExit>, while_check=0x56a880 <CCTKi_ScheduleCallWhile>, if_check=0x56a530 <CCTKi_ScheduleCallIf>, function_process=<optimized out>, data=0x7fffffffd340) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/schedule/ScheduleTraverse.c:158 #16 0x00000000005707db in ScheduleTraverse (CallFunction=0x17addc0 <Carpet::CallFunction(void*, cFunctionData*, void*)>, GH=0x78d3360, where=0x37267a4 "CCTK_RECOVER_VARIABLES") at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:1360 #17 CCTK_ScheduleTraverse (where=0x37267a4 "CCTK_RECOVER_VARIABLES", GH=0x78d3360, CallFunction=0x17addc0 <Carpet::CallFunction(void*, cFunctionData*, void*)>) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/ScheduleInterface.c:891 #18 0x0000000001738d11 in Carpet::ScheduleTraverse (name=0x37267a4 "CCTK_RECOVER_VARIABLES", cctkGH=0x78d3360, where=<optimized out>) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/Initialise.cc:1277 #19 0x00000000017396d9 in CallRecoverVariables (cctkGH=0x78d3360) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/Initialise.cc:241 #20 Carpet::Initialise (fc=0x7fffffffd7e0) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/arrangements/Carpet/Carpet/src/Initialise.cc:115 #21 0x00000000005633a3 in main (argc=2, argv=0x7fffffffd8e8) at /mnt/data/rhaas/postdoc/gr/ET_Lovelace/src/main/flesh.cc:80 }}}
this is revision 1566 of AHFinderDirect and Carpet hash 3555:12ee1105096a attached are the parameter files (ending in -admbase) and stdout as well as stderr. This is with one MPI process and one thread. On my workstation at Caltech.
The inability to solve via a parameter setting is thus mostly related to some parameters not being steerable upon recovery even though they could be changed at least in some ways without problem.