[Users] Problems with regridding?

Geraint Pratten g.pratten at sussex.ac.uk
Wed Mar 2 11:19:30 CST 2016


Hi,

So I've tried to change a few things to see if there was an obvious
solution and there does not seem to be any improvement at all. I've added a
Carpet::poison_value of 100 to the .par file. I have also tried using
various different grid structures (including the
​example
structure used in the /par/arxiv/bbh folders - though adapted to 3D without
symmetry).
​ The errors ​always seem to occur around 75-85M irrespective of the grid
structure though. I really am baffled as to what could be causing this
problem.

The first encounter of a NaN is in TrK


*WARNING level 1 from host i24r03c05s08.sm.lrz.de
<http://i24r03c05s08.sm.lrz.de> process 225*


*    while executing schedule bin NaNChecker_NaNCheck, routine
NaNChecker::NaNChecker_NaNCheck_Check    in thorn NaNChecker, file
/gpfs/work/pr92bo/di25maz/Cactus/arrangements/CactusUtils/NaNChecker/src/NaNCheck.cc:567:
       -> NaN caught in variable 'ML_BSSN::trK' at index (62, 180, 182)
level 0 map 0 with coordinates (-63, 55, 57) in iteration 44160*

with a subsequent pile of NaNs in all the variables at levels 0 and 1 (map
0) at similar coordinates. E.g.

*WARNING level 1 from host i24r01c04s09.sm.lrz.de
<http://i24r01c04s09.sm.lrz.de> process 41*


*   while executing schedule bin NaNChecker_NaNCheck, routine
NaNChecker::NaNChecker_NaNCheck_Check    in thorn NaNChecker, file
/gpfs/work/pr92bo/di25maz/Cactus/arrangements/CactusUtils/NaNChecker/src/NaNCheck.cc:567:
      -> Inf caught in variable 'ADMBASE::gxy' at index (121, 357, 358)
level 1 map 0 with coordinates (-62, 56, 56.5) in iteration 44160*

There are plots from ml_admconstraints-ml_ham.file_225.h5 for the first 4
levels (0-3) at iteration 40960 (by this point the Hamiltonian constraint
is on its way to blowing up) available at:

https://github.com/GeraintPratten/Einstein-Toolkit-Plots

You can clearly see that on level 1 something bizarre is happening, it
looks like there are some real boundary issues. I have no idea what could
be causing this or why.

Thanks again for any feedback!
Geraint

On 26 February 2016 at 07:38, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

>
> On 24 Feb 2016, at 14:51, Geraint Pratten <g.pratten at sussex.ac.uk> wrote:
>
> Hi everyone,
>
> I've been playing around with grids under the assumption that there is no
> reflection symmetry and I seem to be hitting a problem when Carpet goes to
> regrid. The system evolves for ~ 80M quite happily but it hits a point
> where it attempts to regrid levels 1 and up causing all sorts of havoc,
> notably the measure of the Hamiltonian constraint blows up until the
> simulation gracefully terminates due to NaNs.
>
> I've attached a portion of the output for the simulation where it is
> initially evolving quite happily and then the simulation regrids (line
> 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H
>  maximum, this eventually just goes off to ~ 10^{230} and dies).
>
>
> Hi Geraint,
>
> I also have not seen exactly this before.
>
> 10^230 looks like a typical poison value, but in the parameter file you
> sent, you don't set the poison value, so it should default to NaN.  Is the
> file you sent exactly the same as the one you ran?  Can you look at the 2D
> output, for example of H, and check that it is blowing up at this
> refinement boundary?
>
> Have you successfully run similar setups before?  If so, what has changed
> between a successful run and this one?
>
> What version of the ET are you using?
>
>
>
> I'm presuming that when it tries to regrid levels 1 and up it's
> introducing some bad data along the way?
>
> The other issue that I'm not completely sure on is related to the warning
> being thrown up by QuasiLocalMeasures:
>
> WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123:
>   -> Did not manage to find an eigenvector with the eigenvalue 1
> WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
>   -> Integration path starts out too steep
> WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
>   -> Integration path starts out too steep
> WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
>   -> Integration path starts out too steep
> WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
>   -> Integration path starts out too steep
> WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
>   -> Integration path starts out too steep
> WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0
>   while executing schedule bin CCTK_ANALYSIS, routine
> QuasiLocalMeasures::qlm_calculate
>   in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96:
>   -> Integration path starts out too steep
>
> Can anyone help with this or has anyone seen similar issues before?
> Something is clearly not set up correctly but I can't see where my error
> is. If you need any other information, just let me know!
>
> Thanks in advance! Any help is very much appreciated!
> Geraint
>
> ------------------
>
> For reference, the various Carpet settings that I use are as follows:
>
> #------
> # Grid:
> #------
>
> Time::dtfac                 = 0.25
> MoL::ODE_Method             = "rk4"
> MoL::MoL_Intermediate_Steps = 4
> MoL::MoL_Num_Scratch_Levels = 1
>
> #------------------------------
> # Carpet Coord. Grid 3D
> #------------------------------
> CartGrid3D::type            = "coordbase"
> CartGrid3D::domain          = "full"
> CartGrid3D::avoid_origin    = "no"
>
> #------------------------------
> # Carpet Coord. Base
> #------------------------------
> CoordBase::domainsize       = "minmax"
> CoordBase::spacing          = "gridspacing"  # "gridspacing" or "numcells"
>
> CoordBase::xmin = -614.40
> CoordBase::ymin = -614.40
> CoordBase::zmin = -614.40
> CoordBase::xmax =  614.40
> CoordBase::ymax =  614.40
> CoordBase::zmax =  614.40
> CoordBase::dx   =   15.36  # dx or ncells_x, nx = 80
> CoordBase::dy   =   15.36  # dy or ncells_y, nx = 80
> CoordBase::dz   =   15.36  # dz or ncells_z, nx = 80
>
> CoordBase::boundary_size_x_lower        = 5
> CoordBase::boundary_size_y_lower        = 5
> CoordBase::boundary_size_z_lower        = 5
> CoordBase::boundary_size_x_upper        = 5
> CoordBase::boundary_size_y_upper        = 5
> CoordBase::boundary_size_z_upper        = 5
> CoordBase::boundary_shiftout_x_lower    = 0
> CoordBase::boundary_shiftout_y_lower    = 0
> CoordBase::boundary_shiftout_z_lower    = 0
> CoordBase::boundary_shiftout_x_upper    = 0
> CoordBase::boundary_shiftout_y_upper    = 0
> CoordBase::boundary_shiftout_z_upper    = 0
>
> #------------------------------
> # Driver: Ghost Size
> #------------------------------
> Driver::ghost_size                      = 5
>
> #------------------------------
> # Specify Reflection Symmetries
> #------------------------------
> ReflectionSymmetry::reflection_x   = "no"
> ReflectionSymmetry::reflection_y   = "no"
> ReflectionSymmetry::reflection_z   = "no"
> ReflectionSymmetry::avoid_origin_x = "no"
> ReflectionSymmetry::avoid_origin_y = "no"
> ReflectionSymmetry::avoid_origin_z = "no"
>
>
> #------------------------------
> # General Carpet Parameters
> #------------------------------
> Carpet::domain_from_coordbase    = "yes"
> Carpet::enable_all_storage             = "no"
> Carpet::use_buffer_zones               = "yes"
> Carpet::schedule_barriers              = "no"
>
> Carpet::poison_new_timelevels    = "yes"
> Carpet::check_for_poison             = "no"
> #Carpet::poison_value                  = 113 # git version only!
>
> Carpet::init_3_timelevels        = "no"
> Carpet::init_fill_timelevels       = "yes"
>
> CarpetLib::poison_new_memory            = "yes"
> CarpetLib::poison_value                         = 114
> CarpetLib::check_bboxes                       = "no"
> CarpetLib::interleave_communications  = "yes"
> #CarpetLib::vary_tags                             = "yes" # git version
> only!
> CarpetLib::combine_sends                     = "yes"
> #CarpetLib::reduce_mpi_waitall             = "yes" # git version only!
>
> CarpetInterp::tree_search            = "yes"
> CarpetInterp::check_tree_search = "no"
>
> CarpetRegrid2::freeze_unaligned_levels  = "yes"
> #CarpetRegrid2::snap_to_coarse             = "yes"
> CarpetRegrid2::symmetry_rotating180     = "no"
> CarpetRegrid2::min_fraction                     = 1
>
> # System specific Carpet parameters
> Carpet::time_refinement_factors  = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256,
> 512, 1024]" # Should check this...?
> Carpet::max_refinement_levels     = 12
> Carpet::prolongation_order_space  = 5
> Carpet::prolongation_order_time   = 2
>
> CarpetRegrid2::regrid_every = 64
> CarpetRegrid2::num_centres  = 2
>
> CarpetRegrid2::num_levels_1 = 12
> CarpetRegrid2::position_x_1 = 2.00393
> CarpetRegrid2::radius_1[ 1]         = 307.2
> CarpetRegrid2::radius_1[ 2]         = 230.4
> CarpetRegrid2::radius_1[ 3]         = 153.6
> CarpetRegrid2::radius_1[ 4]         = 105.6
> CarpetRegrid2::radius_1[ 5]         =  38.4
> CarpetRegrid2::radius_1[ 6]         =  14.4
> CarpetRegrid2::radius_1[ 7]         =   4.8
> CarpetRegrid2::radius_1[ 8]         =   2.4
> CarpetRegrid2::radius_1[ 9]         =   1.2
> CarpetRegrid2::radius_1[10]         =   0.6
> CarpetRegrid2::radius_1[11]         =   0.3
> CarpetRegrid2::movement_threshold_1 =   0.075
>
> CarpetRegrid2::num_levels_2 = 12
> CarpetRegrid2::position_x_2 = -6.01178
> CarpetRegrid2::radius_2[ 1]         = 307.2
> CarpetRegrid2::radius_2[ 2]         = 230.4
> CarpetRegrid2::radius_2[ 3]         = 153.6
> CarpetRegrid2::radius_2[ 4]         = 105.6
> CarpetRegrid2::radius_2[ 5]         =  38.4
> CarpetRegrid2::radius_2[ 6]         =  14.4
> CarpetRegrid2::radius_2[ 7]         =   4.8
> CarpetRegrid2::radius_2[ 8]         =   2.4
> CarpetRegrid2::radius_2[ 9]         =   1.2
> CarpetRegrid2::radius_2[10]         =   0.6
> CarpetRegrid2::radius_2[11]         =   0.3
> CarpetRegrid2::movement_threshold_2 =   0.075
>
> CarpetMask::excluded_surface       [0] = 0
> CarpetMask::excluded_surface_factor[0] = 1.0
>
> CarpetMask::excluded_surface       [1] = 1
> CarpetMask::excluded_surface_factor[1] = 1.0
>
> CarpetMask::excluded_surface       [2] = 2
> CarpetMask::excluded_surface_factor[2] = 1.0
>
> CarpetTracker::surface[0] = 0
> CarpetTracker::surface[1] = 1
>
> --
> Geraint Pratten
> Postdoctoral Research Associate
>
> Mobile: +44(0) 7581709282
> E-mail: G.Pratten at sussex.ac.uk
> Skype: geraint.pratten
>
> School of Mathematical and Physical Sciences
> Pevensey 3 Building
> University of Sussex
> Falmer Campus
> Brighton
> BN1 9QH
> United Kingdom
> <Output_Cleaned.out>_______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>
>
> --
> Ian Hinder
> http://members.aei.mpg.de/ianhin
>
>


-- 
Geraint Pratten
Postdoctoral Research Associate

Mobile: +44(0) 7581709282
E-mail: G.Pratten at sussex.ac.uk
Skype: geraint.pratten

School of Mathematical and Physical Sciences
Pevensey 3 Building
University of Sussex
Falmer Campus
Brighton
BN1 9QH
United Kingdom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20160302/2f04803e/attachment-0001.html 


More information about the Users mailing list