Hi everyone,
I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs.
I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H maximum, this eventually just goes off to ~ 10^{230} and dies).
I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?
The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:
WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123: -> Did not manage to find an eigenvector with the eigenvalue 1 WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep
Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information, just let me know!
Thanks in advance! Any help is very much appreciated! Geraint
------------------
For reference, the various Carpet settings that I use are as follows:
#------ # Grid: #------
Time::dtfac = 0.25 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 MoL::MoL_Num_Scratch_Levels = 1
#------------------------------ # Carpet Coord. Grid 3D #------------------------------ CartGrid3D::type = "coordbase" CartGrid3D::domain = "full" CartGrid3D::avoid_origin = "no"
#------------------------------ # Carpet Coord. Base #------------------------------ CoordBase::domainsize = "minmax" CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells"
CoordBase::xmin = -614.40 CoordBase::ymin = -614.40 CoordBase::zmin = -614.40 CoordBase::xmax = 614.40 CoordBase::ymax = 614.40 CoordBase::zmax = 614.40 CoordBase::dx = 15.36 # dx or ncells_x, nx = 80 CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 CoordBase::dz = 15.36 # dz or ncells_z, nx = 80
CoordBase::boundary_size_x_lower = 5 CoordBase::boundary_size_y_lower = 5 CoordBase::boundary_size_z_lower = 5 CoordBase::boundary_size_x_upper = 5 CoordBase::boundary_size_y_upper = 5 CoordBase::boundary_size_z_upper = 5 CoordBase::boundary_shiftout_x_lower = 0 CoordBase::boundary_shiftout_y_lower = 0 CoordBase::boundary_shiftout_z_lower = 0 CoordBase::boundary_shiftout_x_upper = 0 CoordBase::boundary_shiftout_y_upper = 0 CoordBase::boundary_shiftout_z_upper = 0
#------------------------------ # Driver: Ghost Size #------------------------------ Driver::ghost_size = 5
#------------------------------ # Specify Reflection Symmetries #------------------------------ ReflectionSymmetry::reflection_x = "no" ReflectionSymmetry::reflection_y = "no" ReflectionSymmetry::reflection_z = "no" ReflectionSymmetry::avoid_origin_x = "no" ReflectionSymmetry::avoid_origin_y = "no" ReflectionSymmetry::avoid_origin_z = "no"
#------------------------------ # General Carpet Parameters #------------------------------ Carpet::domain_from_coordbase = "yes" Carpet::enable_all_storage = "no" Carpet::use_buffer_zones = "yes" Carpet::schedule_barriers = "no"
Carpet::poison_new_timelevels = "yes" Carpet::check_for_poison = "no" #Carpet::poison_value = 113 # git version only!
Carpet::init_3_timelevels = "no" Carpet::init_fill_timelevels = "yes"
CarpetLib::poison_new_memory = "yes" CarpetLib::poison_value = 114 CarpetLib::check_bboxes = "no" CarpetLib::interleave_communications = "yes" #CarpetLib::vary_tags = "yes" # git version only! CarpetLib::combine_sends = "yes" #CarpetLib::reduce_mpi_waitall = "yes" # git version only!
CarpetInterp::tree_search = "yes" CarpetInterp::check_tree_search = "no"
CarpetRegrid2::freeze_unaligned_levels = "yes" #CarpetRegrid2::snap_to_coarse = "yes" CarpetRegrid2::symmetry_rotating180 = "no" CarpetRegrid2::min_fraction = 1
# System specific Carpet parameters Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...? Carpet::max_refinement_levels = 12 Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2
CarpetRegrid2::regrid_every = 64 CarpetRegrid2::num_centres = 2
CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = 2.00393 CarpetRegrid2::radius_1[ 1] = 307.2 CarpetRegrid2::radius_1[ 2] = 230.4 CarpetRegrid2::radius_1[ 3] = 153.6 CarpetRegrid2::radius_1[ 4] = 105.6 CarpetRegrid2::radius_1[ 5] = 38.4 CarpetRegrid2::radius_1[ 6] = 14.4 CarpetRegrid2::radius_1[ 7] = 4.8 CarpetRegrid2::radius_1[ 8] = 2.4 CarpetRegrid2::radius_1[ 9] = 1.2 CarpetRegrid2::radius_1[10] = 0.6 CarpetRegrid2::radius_1[11] = 0.3 CarpetRegrid2::movement_threshold_1 = 0.075
CarpetRegrid2::num_levels_2 = 12 CarpetRegrid2::position_x_2 = -6.01178 CarpetRegrid2::radius_2[ 1] = 307.2 CarpetRegrid2::radius_2[ 2] = 230.4 CarpetRegrid2::radius_2[ 3] = 153.6 CarpetRegrid2::radius_2[ 4] = 105.6 CarpetRegrid2::radius_2[ 5] = 38.4 CarpetRegrid2::radius_2[ 6] = 14.4 CarpetRegrid2::radius_2[ 7] = 4.8 CarpetRegrid2::radius_2[ 8] = 2.4 CarpetRegrid2::radius_2[ 9] = 1.2 CarpetRegrid2::radius_2[10] = 0.6 CarpetRegrid2::radius_2[11] = 0.3 CarpetRegrid2::movement_threshold_2 = 0.075
CarpetMask::excluded_surface [0] = 0 CarpetMask::excluded_surface_factor[0] = 1.0
CarpetMask::excluded_surface [1] = 1 CarpetMask::excluded_surface_factor[1] = 1.0
CarpetMask::excluded_surface [2] = 2 CarpetMask::excluded_surface_factor[2] = 1.0
CarpetTracker::surface[0] = 0 CarpetTracker::surface[1] = 1