Hi everyone,
I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs.
I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H maximum, this eventually just goes off to ~ 10^{230} and dies).
I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?
The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:
WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123: -> Did not manage to find an eigenvector with the eigenvalue 1 WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep
Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information, just let me know!
Thanks in advance! Any help is very much appreciated! Geraint
------------------
For reference, the various Carpet settings that I use are as follows:
#------ # Grid: #------
Time::dtfac = 0.25 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 MoL::MoL_Num_Scratch_Levels = 1
#------------------------------ # Carpet Coord. Grid 3D #------------------------------ CartGrid3D::type = "coordbase" CartGrid3D::domain = "full" CartGrid3D::avoid_origin = "no"
#------------------------------ # Carpet Coord. Base #------------------------------ CoordBase::domainsize = "minmax" CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells"
CoordBase::xmin = -614.40 CoordBase::ymin = -614.40 CoordBase::zmin = -614.40 CoordBase::xmax = 614.40 CoordBase::ymax = 614.40 CoordBase::zmax = 614.40 CoordBase::dx = 15.36 # dx or ncells_x, nx = 80 CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 CoordBase::dz = 15.36 # dz or ncells_z, nx = 80
CoordBase::boundary_size_x_lower = 5 CoordBase::boundary_size_y_lower = 5 CoordBase::boundary_size_z_lower = 5 CoordBase::boundary_size_x_upper = 5 CoordBase::boundary_size_y_upper = 5 CoordBase::boundary_size_z_upper = 5 CoordBase::boundary_shiftout_x_lower = 0 CoordBase::boundary_shiftout_y_lower = 0 CoordBase::boundary_shiftout_z_lower = 0 CoordBase::boundary_shiftout_x_upper = 0 CoordBase::boundary_shiftout_y_upper = 0 CoordBase::boundary_shiftout_z_upper = 0
#------------------------------ # Driver: Ghost Size #------------------------------ Driver::ghost_size = 5
#------------------------------ # Specify Reflection Symmetries #------------------------------ ReflectionSymmetry::reflection_x = "no" ReflectionSymmetry::reflection_y = "no" ReflectionSymmetry::reflection_z = "no" ReflectionSymmetry::avoid_origin_x = "no" ReflectionSymmetry::avoid_origin_y = "no" ReflectionSymmetry::avoid_origin_z = "no"
#------------------------------ # General Carpet Parameters #------------------------------ Carpet::domain_from_coordbase = "yes" Carpet::enable_all_storage = "no" Carpet::use_buffer_zones = "yes" Carpet::schedule_barriers = "no"
Carpet::poison_new_timelevels = "yes" Carpet::check_for_poison = "no" #Carpet::poison_value = 113 # git version only!
Carpet::init_3_timelevels = "no" Carpet::init_fill_timelevels = "yes"
CarpetLib::poison_new_memory = "yes" CarpetLib::poison_value = 114 CarpetLib::check_bboxes = "no" CarpetLib::interleave_communications = "yes" #CarpetLib::vary_tags = "yes" # git version only! CarpetLib::combine_sends = "yes" #CarpetLib::reduce_mpi_waitall = "yes" # git version only!
CarpetInterp::tree_search = "yes" CarpetInterp::check_tree_search = "no"
CarpetRegrid2::freeze_unaligned_levels = "yes" #CarpetRegrid2::snap_to_coarse = "yes" CarpetRegrid2::symmetry_rotating180 = "no" CarpetRegrid2::min_fraction = 1
# System specific Carpet parameters Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...? Carpet::max_refinement_levels = 12 Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2
CarpetRegrid2::regrid_every = 64 CarpetRegrid2::num_centres = 2
CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = 2.00393 CarpetRegrid2::radius_1[ 1] = 307.2 CarpetRegrid2::radius_1[ 2] = 230.4 CarpetRegrid2::radius_1[ 3] = 153.6 CarpetRegrid2::radius_1[ 4] = 105.6 CarpetRegrid2::radius_1[ 5] = 38.4 CarpetRegrid2::radius_1[ 6] = 14.4 CarpetRegrid2::radius_1[ 7] = 4.8 CarpetRegrid2::radius_1[ 8] = 2.4 CarpetRegrid2::radius_1[ 9] = 1.2 CarpetRegrid2::radius_1[10] = 0.6 CarpetRegrid2::radius_1[11] = 0.3 CarpetRegrid2::movement_threshold_1 = 0.075
CarpetRegrid2::num_levels_2 = 12 CarpetRegrid2::position_x_2 = -6.01178 CarpetRegrid2::radius_2[ 1] = 307.2 CarpetRegrid2::radius_2[ 2] = 230.4 CarpetRegrid2::radius_2[ 3] = 153.6 CarpetRegrid2::radius_2[ 4] = 105.6 CarpetRegrid2::radius_2[ 5] = 38.4 CarpetRegrid2::radius_2[ 6] = 14.4 CarpetRegrid2::radius_2[ 7] = 4.8 CarpetRegrid2::radius_2[ 8] = 2.4 CarpetRegrid2::radius_2[ 9] = 1.2 CarpetRegrid2::radius_2[10] = 0.6 CarpetRegrid2::radius_2[11] = 0.3 CarpetRegrid2::movement_threshold_2 = 0.075
CarpetMask::excluded_surface [0] = 0 CarpetMask::excluded_surface_factor[0] = 1.0
CarpetMask::excluded_surface [1] = 1 CarpetMask::excluded_surface_factor[1] = 1.0
CarpetMask::excluded_surface [2] = 2 CarpetMask::excluded_surface_factor[2] = 1.0
CarpetTracker::surface[0] = 0 CarpetTracker::surface[1] = 1
On Wed, Feb 24, 2016 at 01:51:23PM +0000, Geraint Pratten wrote:
I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs.
I didn't encounter this, but looking at your grid setup I don't see a problem with level 1. It should have enough points to the outer boundary - it is the other levels that are much 'tighter', but don't seem to cause problems. The one difference I see for level 1 is the different time refinement factor you have between level 0 and level 1 (factor 1), and between all other levels (factor 2). This could be a hint as to where to look for the cause, but might also be a red herring. A simple test could be using a factor of 2 there as well. It might introduce other problems, but if this particular problem goes away when that level regrids, it would be another hint.
Also, you say that the constraint shoots up: did you only look at the norms? If so: plot it over the entire domain to see where that happens. It might just be that refinement boundary, giving you another clue towards a solution.
Frank
On 24 Feb 2016, at 14:51, Geraint Pratten g.pratten@sussex.ac.uk wrote:
Hi everyone,
I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs.
I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H maximum, this eventually just goes off to ~ 10^{230} and dies).
Hi Geraint,
I also have not seen exactly this before.
10^230 looks like a typical poison value, but in the parameter file you sent, you don't set the poison value, so it should default to NaN. Is the file you sent exactly the same as the one you ran? Can you look at the 2D output, for example of H, and check that it is blowing up at this refinement boundary?
Have you successfully run similar setups before? If so, what has changed between a successful run and this one?
What version of the ET are you using?
I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?
The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:
WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123: -> Did not manage to find an eigenvector with the eigenvalue 1 WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep
Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information, just let me know!
Thanks in advance! Any help is very much appreciated! Geraint
For reference, the various Carpet settings that I use are as follows:
#------ # Grid: #------
Time::dtfac = 0.25 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 MoL::MoL_Num_Scratch_Levels = 1
#------------------------------ # Carpet Coord. Grid 3D #------------------------------ CartGrid3D::type = "coordbase" CartGrid3D::domain = "full" CartGrid3D::avoid_origin = "no"
#------------------------------ # Carpet Coord. Base #------------------------------ CoordBase::domainsize = "minmax" CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells"
CoordBase::xmin = -614.40 CoordBase::ymin = -614.40 CoordBase::zmin = -614.40 CoordBase::xmax = 614.40 CoordBase::ymax = 614.40 CoordBase::zmax = 614.40 CoordBase::dx = 15.36 # dx or ncells_x, nx = 80 CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 CoordBase::dz = 15.36 # dz or ncells_z, nx = 80
CoordBase::boundary_size_x_lower = 5 CoordBase::boundary_size_y_lower = 5 CoordBase::boundary_size_z_lower = 5 CoordBase::boundary_size_x_upper = 5 CoordBase::boundary_size_y_upper = 5 CoordBase::boundary_size_z_upper = 5 CoordBase::boundary_shiftout_x_lower = 0 CoordBase::boundary_shiftout_y_lower = 0 CoordBase::boundary_shiftout_z_lower = 0 CoordBase::boundary_shiftout_x_upper = 0 CoordBase::boundary_shiftout_y_upper = 0 CoordBase::boundary_shiftout_z_upper = 0
#------------------------------ # Driver: Ghost Size #------------------------------ Driver::ghost_size = 5
#------------------------------ # Specify Reflection Symmetries #------------------------------ ReflectionSymmetry::reflection_x = "no" ReflectionSymmetry::reflection_y = "no" ReflectionSymmetry::reflection_z = "no" ReflectionSymmetry::avoid_origin_x = "no" ReflectionSymmetry::avoid_origin_y = "no" ReflectionSymmetry::avoid_origin_z = "no"
#------------------------------ # General Carpet Parameters #------------------------------ Carpet::domain_from_coordbase = "yes" Carpet::enable_all_storage = "no" Carpet::use_buffer_zones = "yes" Carpet::schedule_barriers = "no"
Carpet::poison_new_timelevels = "yes" Carpet::check_for_poison = "no" #Carpet::poison_value = 113 # git version only!
Carpet::init_3_timelevels = "no" Carpet::init_fill_timelevels = "yes"
CarpetLib::poison_new_memory = "yes" CarpetLib::poison_value = 114 CarpetLib::check_bboxes = "no" CarpetLib::interleave_communications = "yes" #CarpetLib::vary_tags = "yes" # git version only! CarpetLib::combine_sends = "yes" #CarpetLib::reduce_mpi_waitall = "yes" # git version only!
CarpetInterp::tree_search = "yes" CarpetInterp::check_tree_search = "no"
CarpetRegrid2::freeze_unaligned_levels = "yes" #CarpetRegrid2::snap_to_coarse = "yes" CarpetRegrid2::symmetry_rotating180 = "no" CarpetRegrid2::min_fraction = 1
# System specific Carpet parameters Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...? Carpet::max_refinement_levels = 12 Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2
CarpetRegrid2::regrid_every = 64 CarpetRegrid2::num_centres = 2
CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = 2.00393 CarpetRegrid2::radius_1[ 1] = 307.2 CarpetRegrid2::radius_1[ 2] = 230.4 CarpetRegrid2::radius_1[ 3] = 153.6 CarpetRegrid2::radius_1[ 4] = 105.6 CarpetRegrid2::radius_1[ 5] = 38.4 CarpetRegrid2::radius_1[ 6] = 14.4 CarpetRegrid2::radius_1[ 7] = 4.8 CarpetRegrid2::radius_1[ 8] = 2.4 CarpetRegrid2::radius_1[ 9] = 1.2 CarpetRegrid2::radius_1[10] = 0.6 CarpetRegrid2::radius_1[11] = 0.3 CarpetRegrid2::movement_threshold_1 = 0.075
CarpetRegrid2::num_levels_2 = 12 CarpetRegrid2::position_x_2 = -6.01178 CarpetRegrid2::radius_2[ 1] = 307.2 CarpetRegrid2::radius_2[ 2] = 230.4 CarpetRegrid2::radius_2[ 3] = 153.6 CarpetRegrid2::radius_2[ 4] = 105.6 CarpetRegrid2::radius_2[ 5] = 38.4 CarpetRegrid2::radius_2[ 6] = 14.4 CarpetRegrid2::radius_2[ 7] = 4.8 CarpetRegrid2::radius_2[ 8] = 2.4 CarpetRegrid2::radius_2[ 9] = 1.2 CarpetRegrid2::radius_2[10] = 0.6 CarpetRegrid2::radius_2[11] = 0.3 CarpetRegrid2::movement_threshold_2 = 0.075
CarpetMask::excluded_surface [0] = 0 CarpetMask::excluded_surface_factor[0] = 1.0
CarpetMask::excluded_surface [1] = 1 CarpetMask::excluded_surface_factor[1] = 1.0
CarpetMask::excluded_surface [2] = 2 CarpetMask::excluded_surface_factor[2] = 1.0
CarpetTracker::surface[0] = 0 CarpetTracker::surface[1] = 1
-- Geraint Pratten Postdoctoral Research Associate
Mobile: +44(0) 7581709282 E-mail: G.Pratten@sussex.ac.uk Skype: geraint.pratten
School of Mathematical and Physical Sciences Pevensey 3 Building University of Sussex Falmer Campus Brighton BN1 9QH United Kingdom <Output_Cleaned.out>_______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Frank Loeffler -
Geraint Pratten -
Ian Hinder