Hi Ian, Frank,

Thanks for the quick replies! I'll look into submitting a simulation using different time refinement levels. I did only look at the norm, so will have take a look at the 2D data shortly to see if it is indeed blowing up at the refinement boundary.

This is the first time I've tried running a simulation using this particular setup, so can't comment on any differences that may be significant. I am currently using Somerville. I just checked and indeed didn't set the poison value on Carpet, just CarpetLib. I will take a look at using Carpet::poison_value = 100 for now.

​Thanks​!
Geraint

 



On 26 February 2016 at 07:38, Ian Hinder <ian.hinder@aei.mpg.de> wrote:

On 24 Feb 2016, at 14:51, Geraint Pratten <g.pratten@sussex.ac.uk> wrote:

Hi everyone,

I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs. 

I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H  maximum, this eventually just goes off to ~ 10^{230} and dies). 

Hi Geraint,

I also have not seen exactly this before.

10^230 looks like a typical poison value, but in the parameter file you sent, you don't set the poison value, so it should default to NaN.  Is the file you sent exactly the same as the one you ran?  Can you look at the 2D output, for example of H, and check that it is blowing up at this refinement boundary?

Have you successfully run similar setups before?  If so, what has changed between a successful run and this one?

What version of the ET are you using?



I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way?

The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures:

WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123:
  -> Did not manage to find an eigenvector with the eigenvalue 1
WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
  -> Integration path starts out too steep
WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
  -> Integration path starts out too steep
WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
  -> Integration path starts out too steep
WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
  -> Integration path starts out too steep
WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
  -> Integration path starts out too steep
WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
  while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate
  in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
  -> Integration path starts out too steep 

Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information, just let me know!

Thanks in advance! Any help is very much appreciated!
Geraint

------------------

For reference, the various Carpet settings that I use are as follows:

#------
# Grid:
#------

Time::dtfac                 = 0.25
MoL::ODE_Method             = "rk4"
MoL::MoL_Intermediate_Steps = 4
MoL::MoL_Num_Scratch_Levels = 1

#------------------------------
# Carpet Coord. Grid 3D
#------------------------------
CartGrid3D::type            = "coordbase"
CartGrid3D::domain          = "full"
CartGrid3D::avoid_origin    = "no"

#------------------------------
# Carpet Coord. Base
#------------------------------
CoordBase::domainsize       = "minmax"
CoordBase::spacing          = "gridspacing"  # "gridspacing" or "numcells"

CoordBase::xmin = -614.40
CoordBase::ymin = -614.40
CoordBase::zmin = -614.40
CoordBase::xmax =  614.40
CoordBase::ymax =  614.40
CoordBase::zmax =  614.40
CoordBase::dx   =   15.36  # dx or ncells_x, nx = 80
CoordBase::dy   =   15.36  # dy or ncells_y, nx = 80 
CoordBase::dz   =   15.36  # dz or ncells_z, nx = 80

CoordBase::boundary_size_x_lower        = 5
CoordBase::boundary_size_y_lower        = 5
CoordBase::boundary_size_z_lower        = 5
CoordBase::boundary_size_x_upper        = 5
CoordBase::boundary_size_y_upper        = 5
CoordBase::boundary_size_z_upper        = 5
CoordBase::boundary_shiftout_x_lower    = 0
CoordBase::boundary_shiftout_y_lower    = 0
CoordBase::boundary_shiftout_z_lower    = 0
CoordBase::boundary_shiftout_x_upper    = 0
CoordBase::boundary_shiftout_y_upper    = 0
CoordBase::boundary_shiftout_z_upper    = 0

#------------------------------
# Driver: Ghost Size
#------------------------------
Driver::ghost_size                      = 5

#------------------------------
# Specify Reflection Symmetries
#------------------------------
ReflectionSymmetry::reflection_x   = "no"
ReflectionSymmetry::reflection_y   = "no"
ReflectionSymmetry::reflection_z   = "no"
ReflectionSymmetry::avoid_origin_x = "no"
ReflectionSymmetry::avoid_origin_y = "no"
ReflectionSymmetry::avoid_origin_z = "no"


#------------------------------
# General Carpet Parameters
#------------------------------
Carpet::domain_from_coordbase    = "yes"
Carpet::enable_all_storage             = "no"
Carpet::use_buffer_zones               = "yes"
Carpet::schedule_barriers              = "no"

Carpet::poison_new_timelevels    = "yes"
Carpet::check_for_poison             = "no"
#Carpet::poison_value                  = 113 # git version only!

Carpet::init_3_timelevels        = "no"
Carpet::init_fill_timelevels       = "yes"

CarpetLib::poison_new_memory            = "yes"
CarpetLib::poison_value                         = 114
CarpetLib::check_bboxes                       = "no"
CarpetLib::interleave_communications  = "yes"
#CarpetLib::vary_tags                             = "yes" # git version only!
CarpetLib::combine_sends                     = "yes"
#CarpetLib::reduce_mpi_waitall             = "yes" # git version only!

CarpetInterp::tree_search            = "yes"
CarpetInterp::check_tree_search = "no"

CarpetRegrid2::freeze_unaligned_levels  = "yes"
#CarpetRegrid2::snap_to_coarse             = "yes"
CarpetRegrid2::symmetry_rotating180     = "no"
CarpetRegrid2::min_fraction                     = 1

# System specific Carpet parameters
Carpet::time_refinement_factors  = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...?
Carpet::max_refinement_levels     = 12
Carpet::prolongation_order_space  = 5
Carpet::prolongation_order_time   = 2

CarpetRegrid2::regrid_every = 64
CarpetRegrid2::num_centres  = 2

CarpetRegrid2::num_levels_1 = 12
CarpetRegrid2::position_x_1 = 2.00393
CarpetRegrid2::radius_1[ 1]         = 307.2 
CarpetRegrid2::radius_1[ 2]         = 230.4 
CarpetRegrid2::radius_1[ 3]         = 153.6 
CarpetRegrid2::radius_1[ 4]         = 105.6 
CarpetRegrid2::radius_1[ 5]         =  38.4 
CarpetRegrid2::radius_1[ 6]         =  14.4 
CarpetRegrid2::radius_1[ 7]         =   4.8 
CarpetRegrid2::radius_1[ 8]         =   2.4 
CarpetRegrid2::radius_1[ 9]         =   1.2 
CarpetRegrid2::radius_1[10]         =   0.6
CarpetRegrid2::radius_1[11]         =   0.3 
CarpetRegrid2::movement_threshold_1 =   0.075

CarpetRegrid2::num_levels_2 = 12
CarpetRegrid2::position_x_2 = -6.01178
CarpetRegrid2::radius_2[ 1]         = 307.2 
CarpetRegrid2::radius_2[ 2]         = 230.4
CarpetRegrid2::radius_2[ 3]         = 153.6
CarpetRegrid2::radius_2[ 4]         = 105.6
CarpetRegrid2::radius_2[ 5]         =  38.4
CarpetRegrid2::radius_2[ 6]         =  14.4
CarpetRegrid2::radius_2[ 7]         =   4.8
CarpetRegrid2::radius_2[ 8]         =   2.4
CarpetRegrid2::radius_2[ 9]         =   1.2
CarpetRegrid2::radius_2[10]         =   0.6
CarpetRegrid2::radius_2[11]         =   0.3
CarpetRegrid2::movement_threshold_2 =   0.075

CarpetMask::excluded_surface       [0] = 0
CarpetMask::excluded_surface_factor[0] = 1.0

CarpetMask::excluded_surface       [1] = 1
CarpetMask::excluded_surface_factor[1] = 1.0

CarpetMask::excluded_surface       [2] = 2
CarpetMask::excluded_surface_factor[2] = 1.0

CarpetTracker::surface[0] = 0
CarpetTracker::surface[1] = 1

--
Geraint Pratten
Postdoctoral Research Associate

Mobile: +44(0) 7581709282
E-mail: G.Pratten@sussex.ac.uk
Skype: geraint.pratten

School of Mathematical and Physical Sciences
Pevensey 3 Building
University of Sussex
Falmer Campus
Brighton
BN1 9QH
United Kingdom
<Output_Cleaned.out>_______________________________________________
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users





--
Geraint Pratten
Postdoctoral Research Associate

Mobile: +44(0) 7581709282
E-mail: G.Pratten@sussex.ac.uk
Skype: geraint.pratten

School of Mathematical and Physical Sciences
Pevensey 3 Building
University of Sussex
Falmer Campus
Brighton
BN1 9QH
United Kingdom