Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

TwoPunctures data storage
by Alfred Castro 03 Mar '16

03 Mar '16

Hi, I am a master student at UIB, and starting to use the Einstein Toolkit for binary black hole evolutions. I am using the TwoPunctures thorn for solving the constraints. In order to speed up tests of grid structures, I would like to just solve on the spectral grid once, store the data in a file, and reread them for different evolutions (e.g. with different finite difference grids). It looks like this is not supported right now. Has anybody done this already? Thanks, -- Alfred

6 9

Re: [Users] Problems with regridding?
by Geraint Pratten 02 Mar '16

02 Mar '16

Hi, So I've tried to change a few things to see if there was an obvious solution and there does not seem to be any improvement at all. I've added a Carpet::poison_value of 100 to the .par file. I have also tried using various different grid structures (including the example structure used in the /par/arxiv/bbh folders - though adapted to 3D without symmetry). The errors always seem to occur around 75-85M irrespective of the grid structure though. I really am baffled as to what could be causing this problem. The first encounter of a NaN is in TrK *WARNING level 1 from host i24r03c05s08.sm.lrz.de <http://i24r03c05s08.sm.lrz.de> process 225* * while executing schedule bin NaNChecker_NaNCheck, routine NaNChecker::NaNChecker_NaNCheck_Check in thorn NaNChecker, file /gpfs/work/pr92bo/di25maz/Cactus/arrangements/CactusUtils/NaNChecker/src/NaNCheck.cc:567: -> NaN caught in variable 'ML_BSSN::trK' at index (62, 180, 182) level 0 map 0 with coordinates (-63, 55, 57) in iteration 44160* with a subsequent pile of NaNs in all the variables at levels 0 and 1 (map 0) at similar coordinates. E.g. *WARNING level 1 from host i24r01c04s09.sm.lrz.de <http://i24r01c04s09.sm.lrz.de> process 41* * while executing schedule bin NaNChecker_NaNCheck, routine NaNChecker::NaNChecker_NaNCheck_Check in thorn NaNChecker, file /gpfs/work/pr92bo/di25maz/Cactus/arrangements/CactusUtils/NaNChecker/src/NaNCheck.cc:567: -> Inf caught in variable 'ADMBASE::gxy' at index (121, 357, 358) level 1 map 0 with coordinates (-62, 56, 56.5) in iteration 44160* There are plots from ml_admconstraints-ml_ham.file_225.h5 for the first 4 levels (0-3) at iteration 40960 (by this point the Hamiltonian constraint is on its way to blowing up) available at: https://github.com/GeraintPratten/Einstein-Toolkit-Plots You can clearly see that on level 1 something bizarre is happening, it looks like there are some real boundary issues. I have no idea what could be causing this or why. Thanks again for any feedback! Geraint On 26 February 2016 at 07:38, Ian Hinder <ian.hinder(a)aei.mpg.de> wrote: > > On 24 Feb 2016, at 14:51, Geraint Pratten <g.pratten(a)sussex.ac.uk> wrote: > > Hi everyone, > > I've been playing around with grids under the assumption that there is no > reflection symmetry and I seem to be hitting a problem when Carpet goes to > regrid. The system evolves for ~ 80M quite happily but it hits a point > where it attempts to regrid levels 1 and up causing all sorts of havoc, > notably the measure of the Hamiltonian constraint blows up until the > simulation gracefully terminates due to NaNs. > > I've attached a portion of the output for the simulation where it is > initially evolving quite happily and then the simulation regrids (line > 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H > maximum, this eventually just goes off to ~ 10^{230} and dies). > > > Hi Geraint, > > I also have not seen exactly this before. > > 10^230 looks like a typical poison value, but in the parameter file you > sent, you don't set the poison value, so it should default to NaN. Is the > file you sent exactly the same as the one you ran? Can you look at the 2D > output, for example of H, and check that it is blowing up at this > refinement boundary? > > Have you successfully run similar setups before? If so, what has changed > between a successful run and this one? > > What version of the ET are you using? > > > > I'm presuming that when it tries to regrid levels 1 and up it's > introducing some bad data along the way? > > The other issue that I'm not completely sure on is related to the warning > being thrown up by QuasiLocalMeasures: > > WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123: > -> Did not manage to find an eigenvector with the eigenvalue 1 > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > > Can anyone help with this or has anyone seen similar issues before? > Something is clearly not set up correctly but I can't see where my error > is. If you need any other information, just let me know! > > Thanks in advance! Any help is very much appreciated! > Geraint > > ------------------ > > For reference, the various Carpet settings that I use are as follows: > > #------ > # Grid: > #------ > > Time::dtfac = 0.25 > MoL::ODE_Method = "rk4" > MoL::MoL_Intermediate_Steps = 4 > MoL::MoL_Num_Scratch_Levels = 1 > > #------------------------------ > # Carpet Coord. Grid 3D > #------------------------------ > CartGrid3D::type = "coordbase" > CartGrid3D::domain = "full" > CartGrid3D::avoid_origin = "no" > > #------------------------------ > # Carpet Coord. Base > #------------------------------ > CoordBase::domainsize = "minmax" > CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells" > > CoordBase::xmin = -614.40 > CoordBase::ymin = -614.40 > CoordBase::zmin = -614.40 > CoordBase::xmax = 614.40 > CoordBase::ymax = 614.40 > CoordBase::zmax = 614.40 > CoordBase::dx = 15.36 # dx or ncells_x, nx = 80 > CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 > CoordBase::dz = 15.36 # dz or ncells_z, nx = 80 > > CoordBase::boundary_size_x_lower = 5 > CoordBase::boundary_size_y_lower = 5 > CoordBase::boundary_size_z_lower = 5 > CoordBase::boundary_size_x_upper = 5 > CoordBase::boundary_size_y_upper = 5 > CoordBase::boundary_size_z_upper = 5 > CoordBase::boundary_shiftout_x_lower = 0 > CoordBase::boundary_shiftout_y_lower = 0 > CoordBase::boundary_shiftout_z_lower = 0 > CoordBase::boundary_shiftout_x_upper = 0 > CoordBase::boundary_shiftout_y_upper = 0 > CoordBase::boundary_shiftout_z_upper = 0 > > #------------------------------ > # Driver: Ghost Size > #------------------------------ > Driver::ghost_size = 5 > > #------------------------------ > # Specify Reflection Symmetries > #------------------------------ > ReflectionSymmetry::reflection_x = "no" > ReflectionSymmetry::reflection_y = "no" > ReflectionSymmetry::reflection_z = "no" > ReflectionSymmetry::avoid_origin_x = "no" > ReflectionSymmetry::avoid_origin_y = "no" > ReflectionSymmetry::avoid_origin_z = "no" > > > #------------------------------ > # General Carpet Parameters > #------------------------------ > Carpet::domain_from_coordbase = "yes" > Carpet::enable_all_storage = "no" > Carpet::use_buffer_zones = "yes" > Carpet::schedule_barriers = "no" > > Carpet::poison_new_timelevels = "yes" > Carpet::check_for_poison = "no" > #Carpet::poison_value = 113 # git version only! > > Carpet::init_3_timelevels = "no" > Carpet::init_fill_timelevels = "yes" > > CarpetLib::poison_new_memory = "yes" > CarpetLib::poison_value = 114 > CarpetLib::check_bboxes = "no" > CarpetLib::interleave_communications = "yes" > #CarpetLib::vary_tags = "yes" # git version > only! > CarpetLib::combine_sends = "yes" > #CarpetLib::reduce_mpi_waitall = "yes" # git version only! > > CarpetInterp::tree_search = "yes" > CarpetInterp::check_tree_search = "no" > > CarpetRegrid2::freeze_unaligned_levels = "yes" > #CarpetRegrid2::snap_to_coarse = "yes" > CarpetRegrid2::symmetry_rotating180 = "no" > CarpetRegrid2::min_fraction = 1 > > # System specific Carpet parameters > Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, > 512, 1024]" # Should check this...? > Carpet::max_refinement_levels = 12 > Carpet::prolongation_order_space = 5 > Carpet::prolongation_order_time = 2 > > CarpetRegrid2::regrid_every = 64 > CarpetRegrid2::num_centres = 2 > > CarpetRegrid2::num_levels_1 = 12 > CarpetRegrid2::position_x_1 = 2.00393 > CarpetRegrid2::radius_1[ 1] = 307.2 > CarpetRegrid2::radius_1[ 2] = 230.4 > CarpetRegrid2::radius_1[ 3] = 153.6 > CarpetRegrid2::radius_1[ 4] = 105.6 > CarpetRegrid2::radius_1[ 5] = 38.4 > CarpetRegrid2::radius_1[ 6] = 14.4 > CarpetRegrid2::radius_1[ 7] = 4.8 > CarpetRegrid2::radius_1[ 8] = 2.4 > CarpetRegrid2::radius_1[ 9] = 1.2 > CarpetRegrid2::radius_1[10] = 0.6 > CarpetRegrid2::radius_1[11] = 0.3 > CarpetRegrid2::movement_threshold_1 = 0.075 > > CarpetRegrid2::num_levels_2 = 12 > CarpetRegrid2::position_x_2 = -6.01178 > CarpetRegrid2::radius_2[ 1] = 307.2 > CarpetRegrid2::radius_2[ 2] = 230.4 > CarpetRegrid2::radius_2[ 3] = 153.6 > CarpetRegrid2::radius_2[ 4] = 105.6 > CarpetRegrid2::radius_2[ 5] = 38.4 > CarpetRegrid2::radius_2[ 6] = 14.4 > CarpetRegrid2::radius_2[ 7] = 4.8 > CarpetRegrid2::radius_2[ 8] = 2.4 > CarpetRegrid2::radius_2[ 9] = 1.2 > CarpetRegrid2::radius_2[10] = 0.6 > CarpetRegrid2::radius_2[11] = 0.3 > CarpetRegrid2::movement_threshold_2 = 0.075 > > CarpetMask::excluded_surface [0] = 0 > CarpetMask::excluded_surface_factor[0] = 1.0 > > CarpetMask::excluded_surface [1] = 1 > CarpetMask::excluded_surface_factor[1] = 1.0 > > CarpetMask::excluded_surface [2] = 2 > CarpetMask::excluded_surface_factor[2] = 1.0 > > CarpetTracker::surface[0] = 0 > CarpetTracker::surface[1] = 1 > > -- > Geraint Pratten > Postdoctoral Research Associate > > Mobile: +44(0) 7581709282 > E-mail: G.Pratten(a)sussex.ac.uk > Skype: geraint.pratten > > School of Mathematical and Physical Sciences > Pevensey 3 Building > University of Sussex > Falmer Campus > Brighton > BN1 9QH > United Kingdom > <Output_Cleaned.out>_______________________________________________ > Users mailing list > Users(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > -- > Ian Hinder > http://members.aei.mpg.de/ianhin > > -- Geraint Pratten Postdoctoral Research Associate Mobile: +44(0) 7581709282 E-mail: G.Pratten(a)sussex.ac.uk Skype: geraint.pratten School of Mathematical and Physical Sciences Pevensey 3 Building University of Sussex Falmer Campus Brighton BN1 9QH United Kingdom

2 1

External library detection script in Perl
by Erik Schnetter 29 Feb '16

29 Feb '16

As discussed earlier today in the telecon, I rewrite the detection (but not the build) script of the FFTW3 package in Perl. I attach the Perl script for your consumption; otherwise, please look at the "perl" branch of this thorn. I am pleasantly surprised by the result. The Perl code was easier to write than the Bash code. The resulting code is also cleaner and easier to read. Support for arrays, scoped variables, and differentiating between "empty string" and "undefined variable" help a lot. Two 20-line Bash helper scripts turned into 2 lines of Perl each. I also took the opportunity to add a few self checks. There were only two minor hiccups, namely finding the equivalents of "mkdir -p" and "date" in Perl. I solved both easily via Google ("make_path", "localtime" / "strftime"). Finally I want to remark that this script is still stand-alone. Translating another script, comparing to Steve's MPI script, and looking at Frank's bash_utils should easily identify common functions that should be put into a module. Steve -- could you as Perl expert do me a favour and scrutinize the script for errors or ways to improve it? -erik -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

2 1

Einstein Toolkit Meeting Reminder
by Frank Loeffler 28 Feb '16

28 Feb '16

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with the URL: https://hangouts.google.com/call/5xt7voshalqmdcztr4zhfeox2ia Frank Löffler

1 0

prolongation, interpolation
by Steven R. Brandt 26 Feb '16

26 Feb '16

I'm looking for a very simple prolongation strategy, if two points exactly coincide the copy the values. If it is between, assign the average of the 2 (or 4) neighboring points. How do I get this behavior? I tried various interpolators (is it "copy"?), some work and some don't. If I try to use "copy", I get an error: There is no vertex-centred stencil for op="LAGRANGE" or op="COPY" with order_space not in {1, 3, 5, 7, 9, 11} I am setting prolongation_order_space = 1, however. I assume it's the vertex centering that's the problem. Thanks. Cheers, Steve

2 1

Re: [Users] Problems with regridding?
by Geraint Pratten 26 Feb '16

26 Feb '16

Hi Ian, Frank, Thanks for the quick replies! I'll look into submitting a simulation using different time refinement levels. I did only look at the norm, so will have take a look at the 2D data shortly to see if it is indeed blowing up at the refinement boundary. This is the first time I've tried running a simulation using this particular setup, so can't comment on any differences that may be significant. I am currently using Somerville. I just checked and indeed didn't set the poison value on Carpet, just CarpetLib. I will take a look at using Carpet::poison_value = 100 for now. Thanks! Geraint On 26 February 2016 at 07:38, Ian Hinder <ian.hinder(a)aei.mpg.de> wrote: > > On 24 Feb 2016, at 14:51, Geraint Pratten <g.pratten(a)sussex.ac.uk> wrote: > > Hi everyone, > > I've been playing around with grids under the assumption that there is no > reflection symmetry and I seem to be hitting a problem when Carpet goes to > regrid. The system evolves for ~ 80M quite happily but it hits a point > where it attempts to regrid levels 1 and up causing all sorts of havoc, > notably the measure of the Hamiltonian constraint blows up until the > simulation gracefully terminates due to NaNs. > > I've attached a portion of the output for the simulation where it is > initially evolving quite happily and then the simulation regrids (line > 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H > maximum, this eventually just goes off to ~ 10^{230} and dies). > > > Hi Geraint, > > I also have not seen exactly this before. > > 10^230 looks like a typical poison value, but in the parameter file you > sent, you don't set the poison value, so it should default to NaN. Is the > file you sent exactly the same as the one you ran? Can you look at the 2D > output, for example of H, and check that it is blowing up at this > refinement boundary? > > Have you successfully run similar setups before? If so, what has changed > between a successful run and this one? > > What version of the ET are you using? > > > > I'm presuming that when it tries to regrid levels 1 and up it's > introducing some bad data along the way? > > The other issue that I'm not completely sure on is related to the warning > being thrown up by QuasiLocalMeasures: > > WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123: > -> Did not manage to find an eigenvector with the eigenvalue 1 > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 > while executing schedule bin CCTK_ANALYSIS, routine > QuasiLocalMeasures::qlm_calculate > in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: > -> Integration path starts out too steep > > Can anyone help with this or has anyone seen similar issues before? > Something is clearly not set up correctly but I can't see where my error > is. If you need any other information, just let me know! > > Thanks in advance! Any help is very much appreciated! > Geraint > > ------------------ > > For reference, the various Carpet settings that I use are as follows: > > #------ > # Grid: > #------ > > Time::dtfac = 0.25 > MoL::ODE_Method = "rk4" > MoL::MoL_Intermediate_Steps = 4 > MoL::MoL_Num_Scratch_Levels = 1 > > #------------------------------ > # Carpet Coord. Grid 3D > #------------------------------ > CartGrid3D::type = "coordbase" > CartGrid3D::domain = "full" > CartGrid3D::avoid_origin = "no" > > #------------------------------ > # Carpet Coord. Base > #------------------------------ > CoordBase::domainsize = "minmax" > CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells" > > CoordBase::xmin = -614.40 > CoordBase::ymin = -614.40 > CoordBase::zmin = -614.40 > CoordBase::xmax = 614.40 > CoordBase::ymax = 614.40 > CoordBase::zmax = 614.40 > CoordBase::dx = 15.36 # dx or ncells_x, nx = 80 > CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 > CoordBase::dz = 15.36 # dz or ncells_z, nx = 80 > > CoordBase::boundary_size_x_lower = 5 > CoordBase::boundary_size_y_lower = 5 > CoordBase::boundary_size_z_lower = 5 > CoordBase::boundary_size_x_upper = 5 > CoordBase::boundary_size_y_upper = 5 > CoordBase::boundary_size_z_upper = 5 > CoordBase::boundary_shiftout_x_lower = 0 > CoordBase::boundary_shiftout_y_lower = 0 > CoordBase::boundary_shiftout_z_lower = 0 > CoordBase::boundary_shiftout_x_upper = 0 > CoordBase::boundary_shiftout_y_upper = 0 > CoordBase::boundary_shiftout_z_upper = 0 > > #------------------------------ > # Driver: Ghost Size > #------------------------------ > Driver::ghost_size = 5 > > #------------------------------ > # Specify Reflection Symmetries > #------------------------------ > ReflectionSymmetry::reflection_x = "no" > ReflectionSymmetry::reflection_y = "no" > ReflectionSymmetry::reflection_z = "no" > ReflectionSymmetry::avoid_origin_x = "no" > ReflectionSymmetry::avoid_origin_y = "no" > ReflectionSymmetry::avoid_origin_z = "no" > > > #------------------------------ > # General Carpet Parameters > #------------------------------ > Carpet::domain_from_coordbase = "yes" > Carpet::enable_all_storage = "no" > Carpet::use_buffer_zones = "yes" > Carpet::schedule_barriers = "no" > > Carpet::poison_new_timelevels = "yes" > Carpet::check_for_poison = "no" > #Carpet::poison_value = 113 # git version only! > > Carpet::init_3_timelevels = "no" > Carpet::init_fill_timelevels = "yes" > > CarpetLib::poison_new_memory = "yes" > CarpetLib::poison_value = 114 > CarpetLib::check_bboxes = "no" > CarpetLib::interleave_communications = "yes" > #CarpetLib::vary_tags = "yes" # git version > only! > CarpetLib::combine_sends = "yes" > #CarpetLib::reduce_mpi_waitall = "yes" # git version only! > > CarpetInterp::tree_search = "yes" > CarpetInterp::check_tree_search = "no" > > CarpetRegrid2::freeze_unaligned_levels = "yes" > #CarpetRegrid2::snap_to_coarse = "yes" > CarpetRegrid2::symmetry_rotating180 = "no" > CarpetRegrid2::min_fraction = 1 > > # System specific Carpet parameters > Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, > 512, 1024]" # Should check this...? > Carpet::max_refinement_levels = 12 > Carpet::prolongation_order_space = 5 > Carpet::prolongation_order_time = 2 > > CarpetRegrid2::regrid_every = 64 > CarpetRegrid2::num_centres = 2 > > CarpetRegrid2::num_levels_1 = 12 > CarpetRegrid2::position_x_1 = 2.00393 > CarpetRegrid2::radius_1[ 1] = 307.2 > CarpetRegrid2::radius_1[ 2] = 230.4 > CarpetRegrid2::radius_1[ 3] = 153.6 > CarpetRegrid2::radius_1[ 4] = 105.6 > CarpetRegrid2::radius_1[ 5] = 38.4 > CarpetRegrid2::radius_1[ 6] = 14.4 > CarpetRegrid2::radius_1[ 7] = 4.8 > CarpetRegrid2::radius_1[ 8] = 2.4 > CarpetRegrid2::radius_1[ 9] = 1.2 > CarpetRegrid2::radius_1[10] = 0.6 > CarpetRegrid2::radius_1[11] = 0.3 > CarpetRegrid2::movement_threshold_1 = 0.075 > > CarpetRegrid2::num_levels_2 = 12 > CarpetRegrid2::position_x_2 = -6.01178 > CarpetRegrid2::radius_2[ 1] = 307.2 > CarpetRegrid2::radius_2[ 2] = 230.4 > CarpetRegrid2::radius_2[ 3] = 153.6 > CarpetRegrid2::radius_2[ 4] = 105.6 > CarpetRegrid2::radius_2[ 5] = 38.4 > CarpetRegrid2::radius_2[ 6] = 14.4 > CarpetRegrid2::radius_2[ 7] = 4.8 > CarpetRegrid2::radius_2[ 8] = 2.4 > CarpetRegrid2::radius_2[ 9] = 1.2 > CarpetRegrid2::radius_2[10] = 0.6 > CarpetRegrid2::radius_2[11] = 0.3 > CarpetRegrid2::movement_threshold_2 = 0.075 > > CarpetMask::excluded_surface [0] = 0 > CarpetMask::excluded_surface_factor[0] = 1.0 > > CarpetMask::excluded_surface [1] = 1 > CarpetMask::excluded_surface_factor[1] = 1.0 > > CarpetMask::excluded_surface [2] = 2 > CarpetMask::excluded_surface_factor[2] = 1.0 > > CarpetTracker::surface[0] = 0 > CarpetTracker::surface[1] = 1 > > -- > Geraint Pratten > Postdoctoral Research Associate > > Mobile: +44(0) 7581709282 > E-mail: G.Pratten(a)sussex.ac.uk > Skype: geraint.pratten > > School of Mathematical and Physical Sciences > Pevensey 3 Building > University of Sussex > Falmer Campus > Brighton > BN1 9QH > United Kingdom > <Output_Cleaned.out>_______________________________________________ > Users mailing list > Users(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > -- > Ian Hinder > http://members.aei.mpg.de/ianhin > > -- Geraint Pratten Postdoctoral Research Associate Mobile: +44(0) 7581709282 E-mail: G.Pratten(a)sussex.ac.uk Skype: geraint.pratten School of Mathematical and Physical Sciences Pevensey 3 Building University of Sussex Falmer Campus Brighton BN1 9QH United Kingdom

1 0

Problems with regridding?
by Geraint Pratten 26 Feb '16

26 Feb '16

Hi everyone, I've been playing around with grids under the assumption that there is no reflection symmetry and I seem to be hitting a problem when Carpet goes to regrid. The system evolves for ~ 80M quite happily but it hits a point where it attempts to regrid levels 1 and up causing all sorts of havoc, notably the measure of the Hamiltonian constraint blows up until the simulation gracefully terminates due to NaNs. I've attached a portion of the output for the simulation where it is initially evolving quite happily and then the simulation regrids (line 1560+) and becomes unstable (see 4th column of line 1689 for *STRAINTS::H maximum, this eventually just goes off to ~ 10^{230} and dies). I'm presuming that when it tries to regrid levels 1 and up it's introducing some bad data along the way? The other issue that I'm not completely sure on is related to the warning being thrown up by QuasiLocalMeasures: WARNING level 3 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_transport.F90:123: -> Did not manage to find an eigenvector with the eigenvalue 1 WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep WARNING level 2 from host i23r01c06s07.sm.lrz.de process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_killing_normalisation.F90:96: -> Integration path starts out too steep Can anyone help with this or has anyone seen similar issues before? Something is clearly not set up correctly but I can't see where my error is. If you need any other information, just let me know! Thanks in advance! Any help is very much appreciated! Geraint ------------------ For reference, the various Carpet settings that I use are as follows: #------ # Grid: #------ Time::dtfac = 0.25 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 MoL::MoL_Num_Scratch_Levels = 1 #------------------------------ # Carpet Coord. Grid 3D #------------------------------ CartGrid3D::type = "coordbase" CartGrid3D::domain = "full" CartGrid3D::avoid_origin = "no" #------------------------------ # Carpet Coord. Base #------------------------------ CoordBase::domainsize = "minmax" CoordBase::spacing = "gridspacing" # "gridspacing" or "numcells" CoordBase::xmin = -614.40 CoordBase::ymin = -614.40 CoordBase::zmin = -614.40 CoordBase::xmax = 614.40 CoordBase::ymax = 614.40 CoordBase::zmax = 614.40 CoordBase::dx = 15.36 # dx or ncells_x, nx = 80 CoordBase::dy = 15.36 # dy or ncells_y, nx = 80 CoordBase::dz = 15.36 # dz or ncells_z, nx = 80 CoordBase::boundary_size_x_lower = 5 CoordBase::boundary_size_y_lower = 5 CoordBase::boundary_size_z_lower = 5 CoordBase::boundary_size_x_upper = 5 CoordBase::boundary_size_y_upper = 5 CoordBase::boundary_size_z_upper = 5 CoordBase::boundary_shiftout_x_lower = 0 CoordBase::boundary_shiftout_y_lower = 0 CoordBase::boundary_shiftout_z_lower = 0 CoordBase::boundary_shiftout_x_upper = 0 CoordBase::boundary_shiftout_y_upper = 0 CoordBase::boundary_shiftout_z_upper = 0 #------------------------------ # Driver: Ghost Size #------------------------------ Driver::ghost_size = 5 #------------------------------ # Specify Reflection Symmetries #------------------------------ ReflectionSymmetry::reflection_x = "no" ReflectionSymmetry::reflection_y = "no" ReflectionSymmetry::reflection_z = "no" ReflectionSymmetry::avoid_origin_x = "no" ReflectionSymmetry::avoid_origin_y = "no" ReflectionSymmetry::avoid_origin_z = "no" #------------------------------ # General Carpet Parameters #------------------------------ Carpet::domain_from_coordbase = "yes" Carpet::enable_all_storage = "no" Carpet::use_buffer_zones = "yes" Carpet::schedule_barriers = "no" Carpet::poison_new_timelevels = "yes" Carpet::check_for_poison = "no" #Carpet::poison_value = 113 # git version only! Carpet::init_3_timelevels = "no" Carpet::init_fill_timelevels = "yes" CarpetLib::poison_new_memory = "yes" CarpetLib::poison_value = 114 CarpetLib::check_bboxes = "no" CarpetLib::interleave_communications = "yes" #CarpetLib::vary_tags = "yes" # git version only! CarpetLib::combine_sends = "yes" #CarpetLib::reduce_mpi_waitall = "yes" # git version only! CarpetInterp::tree_search = "yes" CarpetInterp::check_tree_search = "no" CarpetRegrid2::freeze_unaligned_levels = "yes" #CarpetRegrid2::snap_to_coarse = "yes" CarpetRegrid2::symmetry_rotating180 = "no" CarpetRegrid2::min_fraction = 1 # System specific Carpet parameters Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]" # Should check this...? Carpet::max_refinement_levels = 12 Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2 CarpetRegrid2::regrid_every = 64 CarpetRegrid2::num_centres = 2 CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = 2.00393 CarpetRegrid2::radius_1[ 1] = 307.2 CarpetRegrid2::radius_1[ 2] = 230.4 CarpetRegrid2::radius_1[ 3] = 153.6 CarpetRegrid2::radius_1[ 4] = 105.6 CarpetRegrid2::radius_1[ 5] = 38.4 CarpetRegrid2::radius_1[ 6] = 14.4 CarpetRegrid2::radius_1[ 7] = 4.8 CarpetRegrid2::radius_1[ 8] = 2.4 CarpetRegrid2::radius_1[ 9] = 1.2 CarpetRegrid2::radius_1[10] = 0.6 CarpetRegrid2::radius_1[11] = 0.3 CarpetRegrid2::movement_threshold_1 = 0.075 CarpetRegrid2::num_levels_2 = 12 CarpetRegrid2::position_x_2 = -6.01178 CarpetRegrid2::radius_2[ 1] = 307.2 CarpetRegrid2::radius_2[ 2] = 230.4 CarpetRegrid2::radius_2[ 3] = 153.6 CarpetRegrid2::radius_2[ 4] = 105.6 CarpetRegrid2::radius_2[ 5] = 38.4 CarpetRegrid2::radius_2[ 6] = 14.4 CarpetRegrid2::radius_2[ 7] = 4.8 CarpetRegrid2::radius_2[ 8] = 2.4 CarpetRegrid2::radius_2[ 9] = 1.2 CarpetRegrid2::radius_2[10] = 0.6 CarpetRegrid2::radius_2[11] = 0.3 CarpetRegrid2::movement_threshold_2 = 0.075 CarpetMask::excluded_surface [0] = 0 CarpetMask::excluded_surface_factor[0] = 1.0 CarpetMask::excluded_surface [1] = 1 CarpetMask::excluded_surface_factor[1] = 1.0 CarpetMask::excluded_surface [2] = 2 CarpetMask::excluded_surface_factor[2] = 1.0 CarpetTracker::surface[0] = 0 CarpetTracker::surface[1] = 1 -- Geraint Pratten Postdoctoral Research Associate Mobile: +44(0) 7581709282 E-mail: G.Pratten(a)sussex.ac.uk Skype: geraint.pratten School of Mathematical and Physical Sciences Pevensey 3 Building University of Sussex Falmer Campus Brighton BN1 9QH United Kingdom

3 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 21 Feb '16

21 Feb '16

1 0

ET Compilation Error
by Shawn Rosofsky 15 Feb '16

15 Feb '16

Hello, I recently got ET working on my personal computer (virtual machine with ubuntu using the default ubuntu .cfg file), but I am having issues compiling the code on a local computer cluster using a custom .cfg file based off of the ubuntu .cfg (see end of email). However, when I attempt to compile the code on the cluster, it gets to the point where it begins building the executable, but then it prints out an error as seen below. **Start Code Creating cactus_sim in /n/a3/srosofsky/Cactus/exe from [list of files] /n/a3/srosofsky/Cactus/configs/sim/scratch/external/hwloc/lib/libhwloc.a(topolog\ y-linux.o): In function `hwloc_linux_set_area_membind': /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1224: undefined reference to `mbind' /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1243: undefined reference to `mbind' /n/a3/srosofsky/Cactus/configs/sim/scratch/external/hwloc/lib/libhwloc.a(topolog\ y-linux.o): In function `hwloc_linux_find_kernel_max_numnodes': /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1347: undefined reference to `get_mempolicy' /n/a3/srosofsky/Cactus/configs/sim/scratch/external/hwloc/lib/libhwloc.a(topolog\ y-linux.o): In function `hwloc_linux_get_area_membind': /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1445: undefined reference to `get_mempolicy' /n/a3/srosofsky/Cactus/configs/sim/scratch/external/hwloc/lib/libhwloc.a(topolog\ y-linux.o): In function `hwloc_linux_get_thisthread_membind': /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1392: undefined reference to `get_mempolicy' /n/a3/srosofsky/Cactus/configs/sim/scratch/external/hwloc/lib/libhwloc.a(topolog\ y-linux.o): In function `hwloc_linux_set_thisthread_membind': /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1291: undefined reference to `set_mempolicy' /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1302: undefined reference to `migrate_pages' /n/a3/srosofsky/Cactus/configs/sim/scratch/build/hwloc/hwloc-1.10.1/src/topology\ -linux.c:1314: undefined reference to `set_mempolicy' collect2: ld returned 1 exit status make[1]: *** [/n/a3/srosofsky/Cactus/exe/cactus_sim] Error 1 make: *** [sim] Error 2 **End code The above error occurs when the MPI_DIR in the .cfg file is set to /usr/lib64/openmpi/. I should note that when I change MPI_DIR to BUILD I get a different error message, but still at the same point in the compilation process. **Start Code Creating cactus_sim in /n/a3/srosofsky/Cactus/exe from [file list] /usr/bin/ld: cannot find -lmpi_usempif08 collect2: ld returned 1 exit status make[1]: *** [/n/a3/srosofsky/Cactus/exe/cactus_sim] Error 1 make: *** [sim] Error 2 **End Code Also, I looked at tickets 1717 and 1764 for possible overlap on this issue, but I had difficulty following how the issues in said tickets were resolved. Furthermore, the libnuma-dev and libltdl-dev libraries were installed but the problem still persists. I should note that I am currently using ET version ET_2015_11. Any suggestions would be greatly appreciated. Best, Shawn Here is the configuration file that I used. **Start Code #MPI_DIR=BUILD MPI_DIR = /usr/lib64/openmpi/ PTHREADS_DIR = NO_BUILD HDF5_DIR = BUILD HWLOC_DIR = BUILD HWLOC_EXTRA_LIBS = hwloc **End Code

2 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 13 Feb '16

13 Feb '16

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users