Hi Erik,

it seems to be Carpet or CarpetReduce:

[...] CCTK_POSTSTEP to Dissipation::setup_epsdis INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] SyncGroup "DISSIPATION::EPSDISA_GROUP" iteration=0 time=0 INFO (Carpet): [ml=0][rl=8][tl=0] ProlongateGroups INFO (Carpet): [ml=0][rl=8][tl=0] SyncGroups INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_AllocateMask INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_InitMask INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetExcludedSetup INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetMask::CarpetSurfaceSetup INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Global time local mode call at SetupMask to CarpetReduce::CoordBase_SetupMask INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Global time singlemap mode call at SetupMask to CarpetReduce::CarpetMaskSetup INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Global time local mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_SetMask INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at MaskBase_SetupMaskAll to CarpetReduce::MaskBase_TestMask INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=0][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=0][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=1][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=1][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=2][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=2][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=3][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=3][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=4][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=4][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=5][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=5][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=6][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=6][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=7][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=7][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Entering singlemap mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Entering local mode INFO (Carpet): [ml=0][rl=8][m=0][c=0,lc=0][tl=0] Leaving local mode INFO (Carpet): [ml=0][rl=8][m=0][tl=0] Leaving singlemap mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (CarpetReduce): Simulation domain volume: 512000 INFO (CarpetReduce): Reduction weight sum: 511999.999154925 WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction weight sum differ INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=0][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=0][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=0][tl=0] OutputGH INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=1][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=1][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=1][tl=0] OutputGH INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=2][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=2][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=2][tl=0] OutputGH INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=3][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=3][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=3][tl=0] OutputGH INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=4][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=4][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=4][tl=0] OutputGH INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=5][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=5][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=5][tl=0] OutputGH INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=6][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=6][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=6][tl=0] OutputGH INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Initialisation III at iteration 0 time 0 INFO (Carpet): [ml=0][rl=7][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=7][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=7][tl=0] OutputGH INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Initialisation III at iteration 0 time 0 (global) (meta) INFO (Carpet): [ml=0][rl=8][tl=0] Scheduling CCTK_CPINITIAL INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Meta mode call at CCTK_CPINITIAL to CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Scheduling CCTK_ANALYSIS INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at CCTK_ANALYSIS to CarpetLib::CarpetLib_printtimestats INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at CCTK_ANALYSIS to CarpetLib::CarpetLib_printmemstats Memory statistics from CarpetLib: Current number of objects: 2611.000000 Current allocated memory: 648.773 MB Maximum number of objects: 2629.000 Maximum allocated memory: 652.451 MB Current administrative memory: 2.795 MB Total allocated used system memory: 664.656 MB Total allocated free system memory: 17.647 MB gh::allmemory: 708688 dh::allmemory: 1301208 th::allmemory: 26608 ggf::allmemory: 576000 gdata::allmemory: 182224 INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at CCTK_ANALYSIS to Formaline::Formaline_AnnounceUpdate INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at CCTK_ANALYSIS to IsolatedHorizon::ih_calculate INFO (IsolatedHorizon): Low Spin < 1e-3 ! Skipping coordinate-transformation! INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Meta mode call at CCTK_ANALYSIS to LoopControl::lc_printstats_analysis INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Leaving global mode INFO (Carpet): [tl=0] Entering global mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at CCTK_ANALYSIS to Multipole::Multipole_Calc INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at SH_Extract_Modes to SphericalHarmonics::SH_local_init INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at SH_Extract_Modes to SphericalHarmonics::SH_interp INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at SH_Extract_Modes to SphericalHarmonics::SH_local_sumInt INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=0][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=1][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=2][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=3][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=4][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=5][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=6][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=7][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Entering level mode INFO (Carpet): [ml=0][rl=8][tl=0] Leaving level mode INFO (Carpet): [ml=0][tl=0] Global mode call at SH_Extract_Modes to SphericalHarmonics::SH_sum_patches [node164:8548] *** An error occurred in MPI_Allreduce: the reduction operation MPI_SUM is not defined for non-intrinsic datatypes [node164:8548] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [node164:8548] *** MPI_ERR_OP: invalid reduce operation [node164:8548] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)

Luca

On 28/5/11 9:50 AM, Erik Schnetter wrote:

Luca

I have not seen this error before. It seems to me a configuration problem regarding the MPI version that is used for building or when the job is submitted.

When switching to Mercurial Carpet, you may need to change a few parameters. Most of these are technical changes, e.g. parameters that don't exist any more in Mercurial Carpet. None of these parameter changes lead to MPI problems like this.

To debug this further, it would be important to find out which thorn performs this reduction. Using the parameter Carpet::veryverbose may help here.

-erik

On Fri, May 27, 2011 at 8:02 PM, Luca Baiottibaiotti@ile.osaka-u.ac.jp wrote:

...

Hello,

I am trying to switch to carpet mercurial. I compiled two executables, identical except for the Carpet thorns, one with thorns from the git repository and the other with thorns from the hg repository.

The BBH simulations runs with the git version, but (with the same parfile) it fails with the hg version:

... INFO (CarpetReduce): Simulation domain volume: 512000 INFO (CarpetReduce): Reduction weight sum: 511999.999154925 WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction weight sum differ Memory statistics from CarpetLib: Current number of objects: 2611 Current allocated memory: 649 MB Maximum number of objects: 2.63e+03 Maximum allocated memory: 652 MB Current administrative memory: 2.79 MB Total allocated used system memory: 665 MB Total allocated free system memory: 17.6 MB gh::allmemory: 708688 dh::allmemory: 1301208 th::allmemory: 26608 ggf::allmemory: 576000 gdata::allmemory: 182224 INFO (IsolatedHorizon): Low Spin< 1e-3 ! Skipping coordinate-transformation! [node154:6197] *** An error occurred in MPI_Allreduce: the reduction operation MPI_SUM is not defined for non-intrinsic datat ypes [node154:6197] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [node154:6197] *** MPI_ERR_OP: invalid reduce operation [node154:6197] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)

I am running on damiana, with the same configuration options and qsub scripts for the two executables. When switching from git to hg, should I change, add, or remove some parameter?

Here is a list of the carpet parameters (and a few others) that I set in my parfiles:

ActiveThorns" Carpet CarpetIOScalar CarpetIOASCII CarpetIOBasic CarpetIOHDF5 CarpetInterp CarpetLib CarpetMask CarpetReduce CarpetRegrid2

Parameters:

Carpet::domain_from_coordbase = yes Carpet::max_refinement_levels = 9 CarpetRegrid2::num_centres = 2 CarpetRegrid2::num_levels_1 = 9 CarpetRegrid2::position_x_1 = 4 CarpetRegrid2::radius_1 [1] = 100.0

etc.

Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64]"

Carpet::use_buffer_zones = yes Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2 Carpet::regrid_in_level_mode = yes Carpet::regrid_during_initialisation = no CarpetRegrid2::regrid_every = 256 Carpet::init_fill_timelevels = yes InitBase::initial_data_setup_method = "init_all_levels" CarpetTracker::surface [0] = 0 CarpetTracker::surface [1] = 1 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 Carpet::num_integrator_substeps = 4 MoL::MoL_Num_Scratch_Levels = 1 CarpetMask::excluded_surface [0] = 2 CarpetMask::excluded_surface_factor [0] = 1. CarpetMask::excluded_surface [1] = 3 CarpetMask::excluded_surface [2] = 4 Carpet::poison_new_timelevels = yes Carpet::check_for_poison = no CarpetLib::poison_new_memory = yes CarpetLib::poison_value = 200 Carpet::output_timers_every = 768 CarpetLib::print_timestats_every = 768 CarpetLib::print_memstats_every = 768

Cheers, Luca _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

On 28/5/11 1:38 PM, Erik Schnetter wrote:

On Fri, May 27, 2011 at 9:03 PM, Luca Baiottibaiotti@ile.osaka-u.ac.jp wrote:

...

Hi Erik,

it seems to be Carpet or CarpetReduce:

Luca

Which MPI version is this? Is this a new MPI version? The corresponding code in Carpet hasn't changed in years, except for an added security check.

It is openmpi 1.4.1, in particular, on damiana:

openmpi/1.4.1-intel11.1.064

Luca

On 31 May 2011, at 03:57, Erik Schnetter wrote:

Luca

Try finding out which datatype this is. I suspect this may be about a complex number. If so, I don't know why this is suddenly an error -- as I mentioned before, the code to define the MPI datatype for a Cactus complex number didn't substantially change in the past years.

The corresponding code is in dist.cc in the routine pseudoinit. Maybe we should be using the Fortran complex datatype for this, but Fortran support is not necessarily enabled in MPI (and we don't need it otherwise). Maybe we also need to define our own MPI operators for complex types.

This is happening with the QuasiLocalMeasures test suite when run with the Mercurial version of Carpet on Damiana. It doesn't happen with the Git version (same machine, compiler, optionlist, submit script, etc). The only change to make this fail is to switch from the Git to the Mercurial version of Carpet.

Luca's error message came just after a message from IsolatedHorizon (the thorn from which QuasiLocalMeasures was derived), so it's probably that which is triggering the error.

-erik

On Sat, May 28, 2011 at 5:41 AM, Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:

...

On 28/5/11 1:38 PM, Erik Schnetter wrote:

...

On Fri, May 27, 2011 at 9:03 PM, Luca Baiottibaiotti@ile.osaka-u.ac.jp wrote:

...

Hi Erik,

it seems to be Carpet or CarpetReduce:

Luca

Which MPI version is this? Is this a new MPI version? The corresponding code in Carpet hasn't changed in years, except for an added security check.

It is openmpi 1.4.1, in particular, on damiana:

openmpi/1.4.1-intel11.1.064

Luca

-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Can you find out which call to MPI_Reduce causes this problem? Can you then add print statements before the call to output the arguments? Can you then look in the corresponding mpi.h file to see which datatype etc. is used? If this is not a standard datatype, you could also output the datatypes that Carpet creates (in dist.cc) to see which datatype is actually causing this problem.

-erik

On Wed, Jul 27, 2011 at 4:05 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:

On 28 Jul 2011, at 01:04, Ian Hinder wrote:

...

On 31 May 2011, at 03:57, Erik Schnetter wrote:

...

Luca

Try finding out which datatype this is. I suspect this may be about a complex number. If so, I don't know why this is suddenly an error -- as I mentioned before, the code to define the MPI datatype for a Cactus complex number didn't substantially change in the past years.

The corresponding code is in dist.cc in the routine pseudoinit. Maybe we should be using the Fortran complex datatype for this, but Fortran support is not necessarily enabled in MPI (and we don't need it otherwise). Maybe we also need to define our own MPI operators for complex types.

This is happening with the QuasiLocalMeasures test suite when run with the Mercurial version of Carpet on Damiana.  It doesn't happen with the Git version (same machine, compiler, optionlist, submit script, etc).  The only change to make this fail is to switch from the Git to the Mercurial version of Carpet.

Luca's error message came just after a message from IsolatedHorizon (the thorn from which QuasiLocalMeasures was derived), so it's probably that which is triggering the error.

The test output for one of the tests is at

http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/einsteintoolki...

...

...

-erik

On Sat, May 28, 2011 at 5:41 AM, Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:

...

On 28/5/11 1:38 PM, Erik Schnetter wrote:

...

On Fri, May 27, 2011 at 9:03 PM, Luca Baiottibaiotti@ile.osaka-u.ac.jp wrote:

...

Hi Erik,

it seems to be Carpet or CarpetReduce:

Luca

Which MPI version is this? Is this a new MPI version? The corresponding code in Carpet hasn't changed in years, except for an added security check.

It is openmpi 1.4.1, in particular, on damiana:

openmpi/1.4.1-intel11.1.064

Luca

-- Erik Schnetter schnetter@cct.lsu.edu   http://www.cct.lsu.edu/~eschnett/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Ian Hinder ian.hinder@aei.mpg.de

-- Ian Hinder ian.hinder@aei.mpg.de