On 24 Feb 2016, at 14:51, Geraint Pratten <g.pratten@sussex.ac.uk> wrote:

Hi everyone,

I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs. 

I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H  maximum, this eventually just goes off to ~ 10^{230} and dies). 

Hi Geraint,

I also have not seen exactly this before.

10^230 looks like a typical poison value, but in the parameter file you sent, you don't set the poison value, so it should default to NaN.  Is the file you sent exactly the same as the one you ran?  Can you look at the 2D output, for example of H, and check that it is blowing up at this refinement boundary?

Have you successfully run similar setups before?  If so, what has changed between a successful run and this one?

What version of the ET are you using?

I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?

The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:

WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123:

  -> Did not manage to find an eigenvector with the eigenvalue 1

WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:

  -> Integration path starts out too steep

WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:

  -> Integration path starts out too steep

WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:

  -> Integration path starts out too steep

WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:

  -> Integration path starts out too steep

WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:

  -> Integration path starts out too steep

WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0

  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate

  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:

  -> Integration path starts out too steep 

Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information, just let me know!

Thanks in advance! Any help is very much appreciated!

Geraint

------------------

For reference, the various Carpet settings that I use are as follows:

#------

# Grid:

#------

Time::dtfac                 = 0.25

MoL::ODE_Method             = "rk4"

MoL::MoL_Intermediate_Steps = 4

MoL::MoL_Num_Scratch_Levels = 1

#------------------------------

# Carpet Coord. Grid 3D

#------------------------------

CartGrid3D::type            = "coordbase"

CartGrid3D::domain          = "full"

CartGrid3D::avoid_origin    = "no"

#------------------------------

# Carpet Coord. Base

#------------------------------

CoordBase::domainsize       = "minmax"

CoordBase::spacing          = "gridspacing"  # "gridspacing" or "numcells"

CoordBase::xmin = -614.40

CoordBase::ymin = -614.40

CoordBase::zmin = -614.40

CoordBase::xmax =  614.40

CoordBase::ymax =  614.40

CoordBase::zmax =  614.40

CoordBase::dx   =   15.36  # dx or ncells_x, nx = 80

CoordBase::dy   =   15.36  # dy or ncells_y, nx = 80 

CoordBase::dz   =   15.36  # dz or ncells_z, nx = 80

CoordBase::boundary_size_x_lower        = 5

CoordBase::boundary_size_y_lower        = 5

CoordBase::boundary_size_z_lower        = 5

CoordBase::boundary_size_x_upper        = 5

CoordBase::boundary_size_y_upper        = 5

CoordBase::boundary_size_z_upper        = 5

CoordBase::boundary_shiftout_x_lower    = 0

CoordBase::boundary_shiftout_y_lower    = 0

CoordBase::boundary_shiftout_z_lower    = 0

CoordBase::boundary_shiftout_x_upper    = 0

CoordBase::boundary_shiftout_y_upper    = 0

CoordBase::boundary_shiftout_z_upper    = 0

#------------------------------

# Driver: Ghost Size

#------------------------------

Driver::ghost_size                      = 5

#------------------------------

# Specify Reflection Symmetries

#------------------------------

ReflectionSymmetry::reflection_x   = "no"

ReflectionSymmetry::reflection_y   = "no"

ReflectionSymmetry::reflection_z   = "no"

ReflectionSymmetry::avoid_origin_x = "no"

ReflectionSymmetry::avoid_origin_y = "no"

ReflectionSymmetry::avoid_origin_z = "no"

#------------------------------

# General Carpet Parameters

#------------------------------

Carpet::domain_from_coordbase    = "yes"

Carpet::enable_all_storage             = "no"

Carpet::use_buffer_zones               = "yes"

Carpet::schedule_barriers              = "no"

Carpet::poison_new_timelevels    = "yes"

Carpet::check_for_poison             = "no"

#Carpet::poison_value                  = 113 # git version only!

Carpet::init_3_timelevels        = "no"

Carpet::init_fill_timelevels       = "yes"

CarpetLib::poison_new_memory            = "yes"

CarpetLib::poison_value                         = 114

CarpetLib::check_bboxes                       = "no"

CarpetLib::interleave_communications  = "yes"

#CarpetLib::vary_tags                             = "yes" # git version only!

CarpetLib::combine_sends                     = "yes"

#CarpetLib::reduce_mpi_waitall             = "yes" # git version only!

CarpetInterp::tree_search            = "yes"

CarpetInterp::check_tree_search = "no"

CarpetRegrid2::freeze_unaligned_levels  = "yes"

#CarpetRegrid2::snap_to_coarse             = "yes"

CarpetRegrid2::symmetry_rotating180     = "no"

CarpetRegrid2::min_fraction                     = 1

# System specific Carpet parameters

Carpet::time_refinement_factors  = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...?

Carpet::max_refinement_levels     = 12

Carpet::prolongation_order_space  = 5

Carpet::prolongation_order_time   = 2

CarpetRegrid2::regrid_every = 64

CarpetRegrid2::num_centres  = 2

CarpetRegrid2::num_levels_1 = 12

CarpetRegrid2::position_x_1 = 2.00393

CarpetRegrid2::radius_1[ 1]         = 307.2 

CarpetRegrid2::radius_1[ 2]         = 230.4 

CarpetRegrid2::radius_1[ 3]         = 153.6 

CarpetRegrid2::radius_1[ 4]         = 105.6 

CarpetRegrid2::radius_1[ 5]         =  38.4 

CarpetRegrid2::radius_1[ 6]         =  14.4 

CarpetRegrid2::radius_1[ 7]         =   4.8 

CarpetRegrid2::radius_1[ 8]         =   2.4 

CarpetRegrid2::radius_1[ 9]         =   1.2 

CarpetRegrid2::radius_1[10]         =   0.6

CarpetRegrid2::radius_1[11]         =   0.3 

CarpetRegrid2::movement_threshold_1 =   0.075

CarpetRegrid2::num_levels_2 = 12

CarpetRegrid2::position_x_2 = -6.01178

CarpetRegrid2::radius_2[ 1]         = 307.2 

CarpetRegrid2::radius_2[ 2]         = 230.4

CarpetRegrid2::radius_2[ 3]         = 153.6

CarpetRegrid2::radius_2[ 4]         = 105.6

CarpetRegrid2::radius_2[ 5]         =  38.4

CarpetRegrid2::radius_2[ 6]         =  14.4

CarpetRegrid2::radius_2[ 7]         =   4.8

CarpetRegrid2::radius_2[ 8]         =   2.4

CarpetRegrid2::radius_2[ 9]         =   1.2

CarpetRegrid2::radius_2[10]         =   0.6

CarpetRegrid2::radius_2[11]         =   0.3

CarpetRegrid2::movement_threshold_2 =   0.075

CarpetMask::excluded_surface       [0] = 0

CarpetMask::excluded_surface_factor[0] = 1.0

CarpetMask::excluded_surface       [1] = 1

CarpetMask::excluded_surface_factor[1] = 1.0

CarpetMask::excluded_surface       [2] = 2

CarpetMask::excluded_surface_factor[2] = 1.0

CarpetTracker::surface[0] = 0

CarpetTracker::surface[1] = 1

--

Geraint Pratten

Postdoctoral Research Associate

Mobile: +44(0) 7581709282
E-mail: G.Pratten@sussex.ac.uk
Skype: geraint.pratten

School of Mathematical and Physical Sciences
Pevensey 3 Building
University of Sussex
Falmer Campus
Brighton

BN1 9QH
United Kingdom

<Output_Cleaned.out>_______________________________________________
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users

-- 

Ian Hinder

http://members.aei.mpg.de/ianhin