#800: inconsistent boxes during refluxing very late in the run -----------------------+---------------------------------------------------- Reporter: rhaas | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Other | Version: Keywords: Refluxing | -----------------------+---------------------------------------------------- this is the second half of #797. Same parameter files but using the Refluxing thorn. Eventually I get: {{{ srcbbox: ([25248,27424,26272]:[28704,30432,29088]:[64,64,64]/[394,428,410]:[448,475,454]/[55,48,45]/118800) dstbbox: ([28736,27840,26688]:[28736,28096,28992]:[128,128,128]/[224,217,208]:[224,219,226]/[1,3,19]/57) regbbox: ([28736,27840,26688]:[28736,28096,28992]:[128,128,128]/[224,217,208]:[224,219,226]/[1,3,19]/57) srcbbox: ([25248,27424,31328]:[28704,30432,34144]:[64,WARNING level 0 in thorn CarpetLib processor 80 host shc054 (line 174 of /home/rhaas/cactus/Zelmani/arrangements/Carpet/CarpetLib/src/restrict_3d_vc_rf2.cc): -> Internal error: region extent is not contained in array extent srcbbox: ([25248,27424,23712]:[28704,30432,26592]:[64,64,64]/[394,428,370]:[448,475,415]/[55,48,46]/121440) dstbbox: ([28736,27840,24512]:[28736,28096,25792]:[128,128,128]/[224,217,191]:[224,219,201]/[1,3,11]/33) regbbox: ([28736,27840,24512]:[28736,28096,25792]:[128,128,128]/[224,217,191]:[224,219,201]/[1,3,11]/33) 64,64]/[394,428,489]:[448,475,533]/[55,48,45]/118800) dstbbox: ([28736,27840,31680]:[28736,28096,33344]:[128,128,128]/[224,217,247]:[224,219,260]/[1,3,14]/42) regbbox: ([28736,27840,31680]:[28736,28096,33344]:[128,128,128]/[224,217,247]:[224,219,260]/[1,3,14]/42) WARNING level 0 in thorn CarpetLib processor 82 host shc053 (line 174 of /home/rhaas/cactus/Zelmani/arrangements/Carpet/CarpetLib/src/restrict_3d_vc_rf2.cc): -> Internal error: region extent is not contained in array extent WARNING level 0 in thorn CarpetLib processor 79 host shc055 (line 174 of /home/rhaas/cactus/Zelmani/arrangements/Carpet/CarpetLib/src/restrict_3d_vc_rf2.cc): -> Internal error: region extent is not contained in array extent srcbbox: ([25248,27424,28768]:[28704,30432,31648]:[64,64,64]/[394,428,449]:[448,475,494]/[55,48,46]/121440) dstbbox: ([28736,27840,29120]:[28736,28096,29888]:[128,128,128]/[224,217,227]:[224,219,233]/[1,3,7]/21) regbbox: ([28736,27840,29120]:[28736,28096,29888]:[128,128,128]/[224,217,227]:[224,219,233]/[1,3,7]/21) WARNING level 0 in thorn CarpetLib processor 81 host shc054 (line 174 of /home/rhaas/cactus/Zelmani/arrangements/Carpet/CarpetLib/src/restrict_3d_vc_rf2.cc): -> Internal error: region extent is not contained in array extent }}}
which happens during refluxing
{{{ INFO (Carpet): [ml=0][rl=2][tl=0] Evolution/PostRestrict at iteration 83712 time 196.2 INFO (Carpet): [ml=0][rl=2][tl=0] Scheduling CCTK_POSTRESTRICT INFO (Carpet): [ml=0][rl=2][tl=0] Level mode call at CCTK_POSTRESTRICT to Refluxing::Refluxing_CorrectState INFO (Refluxing): Refluxing at iteration 83712 on level 2 of 4 INFO (Refluxing): Refluxing on level 2: INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (CarpetLib): About to MPI_Isend to processor 1 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 2 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 3 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 7 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 8 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 9 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Waitall INFO (CarpetLib): Finished MPI_Waitall INFO (CarpetLib): About to MPI_Waitall INFO (CarpetLib): Finished MPI_Waitall INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Refluxing): Refluxing on level 2 map 0 component 0 direction 0 face 0: [25,25,25]:[25,31,35] INFO (Refluxing): Refluxing on level 2 map 0 component 0 direction 1 face 0: [25,25,25]:[33,25,35] INFO (Refluxing): Refluxing on level 2 map 0 component 0 direction 2 face 0: [25,25,25]:[33,31,25] INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "GRHYDRO::DENS" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "GRHYDRO::SCON" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "GRHYDRO::TAU" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] ProlongateGroups INFO (CarpetLib): About to MPI_Irecv from processor 1 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 6 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 36 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 39 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 48 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Waitall INFO (CarpetLib): Finished MPI_Waitall INFO (CarpetLib): About to MPI_Waitall INFO (CarpetLib): Finished MPI_Waitall INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroups INFO (CarpetLib): About to MPI_Irecv from processor 1 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 2 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 3 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 7 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 8 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 9 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 10 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Isend to processor 1 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 2 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 3 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 7 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 8 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 9 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Isend to processor 10 for type double INFO (CarpetLib): Finished MPI_Isend INFO (CarpetLib): About to MPI_Waitall INFO (CarpetLib): Finished MPI_Waitall INFO (CarpetLib): About to MPI_Waitall INFO (CarpetLib): Finished MPI_Waitall INFO (Carpet): [ml=0][rl=2][tl=0] Level mode call at MoL_PostStep to ADMBase::ADMBase_Boundaries INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "ADMBASE::LAPSE" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "ADMBASE::DTLAPSE" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "ADMBASE::SHIFT" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "ADMBASE::DTSHIFT" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "ADMBASE::METRIC" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] SyncGroup "ADMBASE::CURV" iteration=83712 time=196.2 INFO (Carpet): [ml=0][rl=2][tl=0] ProlongateGroups INFO (CarpetLib): About to MPI_Irecv from processor 1 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 6 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 36 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 39 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Irecv from processor 48 for type double INFO (CarpetLib): Finished MPI_Irecv INFO (CarpetLib): About to MPI_Waitall }}}
The full stdout and stderr files are about 250MB for this (because of all the verbosity).
I agree that this is not a very helpful error report. Unfortunately I don't know how to distill more useful information out of it :-(