[Users] Llama - Interpatch boundaries found
Samuel Tootle
tootle at itp.uni-frankfurt.de
Mon Jan 16 10:30:07 CST 2023
All:
I've been playing with the GW150914 gallery file and testing the setup
with different mass ratio BBH initial data given how fast the
simulations are using the multipatch setup. I made some modifications
to the gallery script to match my setup (the final par is attached),
however, I keep getting an odd error:
> INFO (Coordinates): Setting up global coordinates for thornburg04.
> INFO (Coordinates): Setting local coordinates: Thornburg04
> WARNING level 0 from host r36c1t1n1.ib0.hawk.hww.hlrs.de process 0
> while executing schedule bin Interpolate2Test, routine
> Interpolate2::Interpolate2TestInit
> in thorn Interpolate2, file
> /zhome/academic/HLRS/xfp/xfpstoot/lib/FreeTHC2020_new/Cactus/arrangements/Llama/Interpolate2/src/test.cc:77:
> -> Refinement level 1 contains inter-patch boundaries; this is
> currently not supported
The current setup is looking at a BBH setup with q = 4. I had the same
problem with q=2 as it seemed some of the refinement levels from the
primary came close to one of those from the secondary so I staggered
them more by making the refinement levels for the primary increase by
factors of two and a factor of 1.5 for the secondary. However, I am
getting the same error even when using this new staggering setup.
Looking at the grid output (see run.log) I don't see where this
"inter-patch" issue is showing up and increasing verbosity didn't shed
further light from my optic. Before dedicating more time, I wanted to
see if maybe the more experienced with these thorns might have some
insight as to what the problem may be.
Cheers,
Samuel
-------------- next part --------------
################################################################################
# Script variables
################################################################################
# D = 5.0
# M = 5.0
# __file__ = /home/user/phd/References/par/BBH_FUKA.rpar
# __name__ = __main__
# ahrm = 1.0
# ahrp = 4.0
# chim = 0.0
# chip = 0.0
# col_width = 19
# dt0 = 1.1828571428571428
# dt_it = 0.009241071428571428
# dtdx = 0.45
# dtdx0 = 0.225
# e = 2.718281828459045
# expected_merger = 1000.0
# final_time = 3500.0
# find_cah = 0
# h0 = 5.257142857142857
# h0_min = 6.133333333333333
# hfm_min = 0.04791666666666666
# horizon_every = 256
# hr = 5.257142857142857
# hr_min = 9.6
# i = 7
# inf = inf
# key = xp
# levelsm = [0,73.600000,36.800000,18.400000,9.200000,4.600000,2.300000,1.150000]
# levelsp = [0,76.800000,38.400000,19.200000,9.600000,4.800000]
# maxrls = 9
# mm = 1.0
# mp = 4.0
# mpr0 = 500.0
# mpr1 = 575.0
# mpr2 = 680.0
# mpr3 = 835.0
# mpr4 = 1000.0
# mpr5 = 1500.0
# n = 28
# n_angular = 28
# n_min = 24
# nan = nan
# out2d_every = 512
# out3d_every = 0
# out_every = 128
# outermost_detector = 1500.0
# pi = 3.141592653589793
# q = 4.0
# rl0_every = 128
# rl1_every = 128
# rl_max = 8
# rlsm = 8
# rlsp = 6
# rm = 1.15
# rp = 4.8
# sphere_inner_radius = 147.2
# sphere_outer_radius = 4973.257142857143
# tau = 6.283185307179586
# time_after_merger = 1000.0
# val = 5.0
# wave_extract_every = 128
# waveform_length = 2000.0
# xm = -20.0
# xp = 5.0
################################################################################
# Active thorns
################################################################################
ActiveThorns = "
ADMBase
ADMCoupling
ADMMacros
AEILocalInterp
AHFinderDirect
Boundary
Carpet
CarpetIOASCII
CarpetIOBasic
CarpetIOHDF5
CarpetIOScalar
CarpetInterp
CarpetInterp2
CarpetLib
CarpetReduce
CarpetRegrid2
CarpetTracker
CartGrid3D
CoordBase
CoordGauge
Coordinates
CoordinatesSymmetry
#Formaline
GlobalDerivative
hwloc
IOUtil
InitBase
Interpolate2
QuasiLocalMeasures
LocalInterp
LoopControl
MoL
NaNChecker
PunctureTracker
Slab
SpaceMask
SphericalSurface
StaticConformal
SummationByParts
SymBase
SystemStatistics
SystemTopology
TerminationTrigger
TensorTypes
Time
TmunuBase
#TwoPunctures
Vectors
ML_BSSN
ML_BSSN_Helper
NewRad
GenericFD
WeylScal4
Multipole
# WaveExtractCPM
ADMDerivatives
"
################################################################################
# Grid structure
################################################################################
Carpet::domain_from_multipatch = yes
CartGrid3D::type = "multipatch"
CartGrid3D::set_coordinate_ranges_on = "all maps"
Coordinates::coordinate_system = "Thornburg04"
Coordinates::h_cartesian = 5.257142857142857
Coordinates::h_radial = 5.257142857142857
Coordinates::sphere_inner_radius = 147.2
Coordinates::sphere_outer_radius = 4973.257142857143
Coordinates::n_angular = 28
Driver::ghost_size = 5
Coordinates::patch_boundary_size = 5
Coordinates::additional_overlap_size = 3
Coordinates::outer_boundary_size = 5
# change these (to their defaults) to disable z-zymmetry
CoordinatesSymmetry::reflection_z = yes
CoordinatesSymmetry::stagger = no
Coordinates::symmetry = "+z bitant"
Coordinates::additional_symmetry_size = 1
Coordinates::verbose = yes
Time::timestep_method = "given"
Time::timestep = 1.1828571428571428
Carpet::time_refinement_factors = "[1,1,2,4,8,16,32,64,128,256]"
#Carpet::veryverbose = "yes"
################################################################################
# Mesh refinement
################################################################################
Carpet::max_refinement_levels = 9
CarpetRegrid2::num_centres = 2
CarpetRegrid2::num_levels_1 = 6
CarpetRegrid2::position_x_1 = 5.0
CarpetRegrid2::radius_1 = [0,80.800000,38.400000,19.200000,9.600000,4.800000]
CarpetRegrid2::num_levels_2 = 8
CarpetRegrid2::position_x_2 = -20.0
CarpetRegrid2::radius_2 = [0,73.600000,36.400000,18.400000,9.200000,4.600000,2.300000,1.20000]
Carpet::use_buffer_zones = yes
Carpet::prolongation_order_space = 5
Carpet::prolongation_order_time = 2
CarpetRegrid2::regrid_every = 128
CarpetRegrid2::verbose = no
Carpet::grid_coordinates_filename = "carpet-grid.asc"
################################################################################
# Initial data
################################################################################
ActiveThorns ="HydroBase"
HydroBase::timelevels = 3
ADMBase::metric_type = "physical"
ActiveThorns = "Kadath_Importer"
ADMBase::initial_data = "Kadath"
ADMBase::initial_lapse = "Kadath"
ADMBase::initial_shift = "Kadath"
ADMBase::initial_dtlapse = "Kadath"
ADMBase::initial_dtshift = "Kadath"
Kadath_Importer::type = "BBH"
Kadath_Importer::filename = "../id/id.info"
Carpet::init_fill_timelevels = yes
InitBase::initial_data_setup_method = "init_single_level"
################################################################################
# Evolution and boundary
################################################################################
GenericFD::jacobian_group = "Coordinates::jacobian"
GenericFD::jacobian_derivative_group = "Coordinates::jacobian2"
GenericFD::jacobian_identity_map = 0
ADMBase::evolution_method = "ML_BSSN"
ADMBase::lapse_evolution_method = "ML_BSSN"
ADMBase::shift_evolution_method = "ML_BSSN"
ADMBase::dtlapse_evolution_method = "ML_BSSN"
ADMBase::dtshift_evolution_method = "ML_BSSN"
ML_BSSN::fdOrder = 8
# 1+log slicing requires harmonicN = 1 and harmonicF = 2.0
ML_BSSN::harmonicN = 1
ML_BSSN::harmonicF = 2.0
ML_BSSN::ShiftGammaCoeff = 0.75
ML_BSSN::BetaDriver = 1.0
ML_BSSN::advectLapse = 1
ML_BSSN::advectShift = 1
ML_BSSN::MinimumLapse = 1.0e-8
# conformalaMethod = 1 for W, 0 for phi
ML_BSSN::conformalMethod = 1
# We apply dissipation using GlobalDerivaitive so disable it here
ML_BSSN::epsDiss = 0.0
ML_BSSN::dt_lapse_shift_method = "noLapseShiftAdvection"
ML_BSSN::initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::rhs_boundary_condition = "scalar"
Boundary::radpower = 2
################################################################################
# BH tracking
################################################################################
CarpetTracker::surface [0] = 0
CarpetTracker::surface [1] = 1
PunctureTracker::track [0] = yes
PunctureTracker::initial_x [0] = 5.0
PunctureTracker::which_surface_to_store_info[0] = 0
PunctureTracker::track [1] = yes
PunctureTracker::initial_x [1] = -20.0
PunctureTracker::which_surface_to_store_info[1] = 1
################################################################################
# Spatial finite differencing
################################################################################
SummationByParts::order = 8
# Drop order instead of using upwinded stencils, only for advection derivatives
SummationByParts::sbp_upwind_deriv = no
SummationByParts::sbp_1st_deriv = yes
SummationByParts::sbp_2nd_deriv = no
SummationByParts::onesided_interpatch_boundaries = no
SummationByParts::onesided_outer_boundaries = yes
SummationByParts::use_dissipation = no
GlobalDerivative::use_dissipation = yes
SummationByParts::scale_with_h = yes
SummationByParts::dissipation_type = "Kreiss-Oliger"
SummationByParts::epsdis = 0.15
# Because dt/dx is half that of the other levels we also need half the dissipation
GlobalDerivative::epsdis_for_level [0] = 0.075
# Variables for dissipation
SummationByParts::vars = "
ML_BSSN::ML_log_confac
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
"
################################################################################
# Time integration
################################################################################
MoL::ODE_Method = "rk4"
MoL::MoL_Intermediate_Steps = 4
MoL::MoL_Num_Scratch_Levels = 1
################################################################################
# Interpolation
################################################################################
CarpetInterp::check_tree_search = no
CarpetInterp::tree_search = yes
# Use 5-th order interpatch interpolation on the Llama grid
Interpolate::interpolator_order = 5
################################################################################
# Psi4 computation
################################################################################
WeylScal4::fdOrder = 8
WeylScal4::calc_scalars = "psis"
WeylScal4::calc_invariants = "always"
################################################################################
# Psi4 mode decomposition
################################################################################
# Radii are chosen to be evenly spaced in 1/r as that is the variable
# extrapolation is performed in
Multipole::nradii = 7
Multipole::radius[0] = 500.0
Multipole::radius[1] = 575.0
Multipole::radius[2] = 680.0
Multipole::radius[3] = 835.0
Multipole::radius[4] = 1000.0
Multipole::radius[5] = 1500.0
# Multipole::radius[6] = mpr6
Multipole::ntheta = 120
Multipole::nphi = 240
Multipole::variables = "WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='psi4'}"
Multipole::out_every = 128
Multipole::l_max = 8
Multipole::output_hdf5 = yes
# Disable ASCII output to avoid creating a large number of files
Multipole::output_ascii = no
################################################################################
# Gauge invariant perturbations of Schwarzschild (ZM-CPM variables)
################################################################################
# WaveExtractCPM::out_every = 128
# WaveExtractCPM::use_carpetinterp2 = no
# WaveExtractCPM::calc_when_necessary = no
# WaveExtractCPM::verbose = 0
# WaveExtractCPM::maximum_detector_number = 7
# WaveExtractCPM::switch_output_format = 100
# WaveExtractCPM::rsch2_computation = "average Schwarzschild metric"
# WaveExtractCPM::l_mode = 8
# WaveExtractCPM::m_mode = 8
# WaveExtractCPM::detector_radius [0] = 500.0
# WaveExtractCPM::detector_radius [1] = 575.0
# WaveExtractCPM::detector_radius [2] = 680.0
# WaveExtractCPM::detector_radius [3] = 835.0
# WaveExtractCPM::detector_radius [4] = 1000.0
# WaveExtractCPM::detector_radius [5] = 1500.0
# WaveExtractCPM::detector_radius [6] = mpr6
# WaveExtractCPM::maxntheta = 120
# WaveExtractCPM::maxnphi = 240
# WaveExtractCPM::output_hdf5 = yes
# WaveExtractCPM::output_ascii = no
# WaveExtractCPM::output_h = yes
# WaveExtractCPM::output_Psi = yes
################################################################################
# Apparent Horizons
################################################################################
AHFinderDirect::N_horizons = 3
AHFinderDirect::find_every = 256
AHFinderDirect::output_h_every = 0
AHFinderDirect::max_Newton_iterations__initial = 50
AHFinderDirect::max_Newton_iterations__subsequent = 50
AHFinderDirect::max_allowable_Theta_growth_iterations = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars = "order=4"
AHFinderDirect::surface_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars = "order=4"
AHFinderDirect::verbose_level = "physics details"
AHFinderDirect::move_origins = yes
AHFinderDirect::origin_x [1] = 5.0
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = 5.0
AHFinderDirect::initial_guess__coord_sphere__radius [1] = 4.8
AHFinderDirect::which_surface_to_store_info [1] = 2
AHFinderDirect::set_mask_for_individual_horizon [1] = no
AHFinderDirect::reset_horizon_after_not_finding [1] = no
AHFinderDirect::track_origin_from_grid_scalar [1] = yes
AHFinderDirect::track_origin_source_x [1] = "PunctureTracker::pt_loc_x[0]"
AHFinderDirect::track_origin_source_y [1] = "PunctureTracker::pt_loc_y[0]"
AHFinderDirect::track_origin_source_z [1] = "PunctureTracker::pt_loc_z[0]"
AHFinderDirect::max_allowable_horizon_radius [1] = 3
AHFinderDirect::origin_x [2] = -20.0
AHFinderDirect::initial_guess__coord_sphere__x_center[2] = -20.0
AHFinderDirect::initial_guess__coord_sphere__radius [2] = 1.15
AHFinderDirect::which_surface_to_store_info [2] = 3
AHFinderDirect::set_mask_for_individual_horizon [2] = no
AHFinderDirect::reset_horizon_after_not_finding [2] = no
AHFinderDirect::track_origin_from_grid_scalar [2] = yes
AHFinderDirect::track_origin_source_x [2] = "PunctureTracker::pt_loc_x[1]"
AHFinderDirect::track_origin_source_y [2] = "PunctureTracker::pt_loc_y[1]"
AHFinderDirect::track_origin_source_z [2] = "PunctureTracker::pt_loc_z[1]"
AHFinderDirect::max_allowable_horizon_radius [2] = 3
AHFinderDirect::origin_x [3] = 0
AHFinderDirect::find_after_individual [3] = 100
AHFinderDirect::initial_guess__coord_sphere__x_center[3] = 0
AHFinderDirect::initial_guess__coord_sphere__radius [3] =6
AHFinderDirect::which_surface_to_store_info [3] = 4
AHFinderDirect::set_mask_for_individual_horizon [3] = no
AHFinderDirect::max_allowable_horizon_radius [3] = 6
################################################################################
# Spherical surfaces
################################################################################
SphericalSurface::nsurfaces = 5
SphericalSurface::maxntheta = 66
SphericalSurface::maxnphi = 124
SphericalSurface::verbose = no
# Surfaces 0 and 1 are used by PunctureTracker
# Horizon 1
SphericalSurface::ntheta [2] = 41
SphericalSurface::nphi [2] = 80
SphericalSurface::nghoststheta [2] = 2
SphericalSurface::nghostsphi [2] = 2
# Horizon 2
SphericalSurface::ntheta [3] = 41
SphericalSurface::nphi [3] = 80
SphericalSurface::nghoststheta [3] = 2
SphericalSurface::nghostsphi [3] = 2
# Common horizon
SphericalSurface::ntheta [4] = 41
SphericalSurface::nphi [4] = 80
SphericalSurface::nghoststheta [4] = 2
SphericalSurface::nghostsphi [4] = 2
################################################################################
# Isolated Horizons
################################################################################
QuasiLocalMeasures::verbose = no
QuasiLocalMeasures::veryverbose = no
QuasiLocalMeasures::interpolator = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options = "order=4"
QuasiLocalMeasures::spatial_order = 4
QuasiLocalMeasures::num_surfaces = 3
QuasiLocalMeasures::surface_index [0] = 2
QuasiLocalMeasures::surface_index [1] = 3
QuasiLocalMeasures::surface_index [2] = 4
QuasiLocalMeasures::output_vtk_every = 0
################################################################################
# Correctness checking
################################################################################
Carpet::poison_new_timelevels = no
Carpet::check_for_poison = no
NaNChecker::check_every = 256
NanChecker::check_after = 0
NaNChecker::report_max = 10
NaNChecker::verbose = "all"
NaNChecker::action_if_found = terminate
NaNChecker::out_NaNmask = yes
NaNChecker::check_vars = "
ML_BSSN::ML_log_confac
"
################################################################################
# Timers
################################################################################
Carpet::output_timer_tree_every = 1024
Carpet::output_initialise_timer_tree = yes
################################################################################
# Output
################################################################################
IO::out_dir = "bbhq2"
IOScalar::one_file_per_group = yes
IOASCII::one_file_per_group = yes
IOBasic::outInfo_every = 1
IOBasic::outInfo_reductions = "minimum maximum"
IOBasic::outInfo_vars = "
ML_BSSN::ML_log_confac
Carpet::physical_time_per_hour
SystemStatistics::maxrss_mb
SystemStatistics::swap_used_mb
"
IOScalar::outScalar_every = 256
IOScalar::outScalar_reductions = "minimum maximum average"
IOScalar::outScalar_vars = "SystemStatistics::process_memory_mb"
IOASCII::out0D_every = 256
IOASCII::out0D_vars = "
Carpet::timing
PunctureTracker::pt_loc
QuasiLocalMeasures::qlm_scalars{out_every = 256}
"
IOASCII::out1D_every = 0
IOASCII::out1D_d = no
IOASCII::out1D_x = yes
IOASCII::out1D_y = no
IOASCII::out1D_z = yes
IOASCII::out1D_vars = "
ML_BSSN::ML_log_confac
ML_BSSN::ML_trace_curv
WeylScal4::Psi4r
"
IOASCII::out2D_every = 0
IOASCII::out2D_vars = "
"
IOHDF5::out_every = 0
IOHDF5::out_vars = "
Grid::Coordinates{out_every=1000000000 refinement_levels={0}}
ML_BSSN::ML_log_confac
WeylScal4::Psi4r
WeylScal4::Psi4i
WeylScal4::curvIr{refinement_levels={3 5}}
WeylScal4::curvIi{refinement_levels={3 5}}
WeylScal4::curvJr{refinement_levels={3 5}}
WeylScal4::curvJi{refinement_levels={3 5}}
"
IOHDF5::out2D_every = 512
IOHDF5::out2D_vars = "
Grid::Coordinates{out_every=1000000000 refinement_levels={0}}
ML_BSSN::alpha
ML_BSSN::phi
ML_BSSN::trK
WeylScal4::Psi4r
WeylScal4::Psi4i
"
################################################################################
# Checkpointing and recovery
################################################################################
CarpetIOHDF5::checkpoint = yes
IO::checkpoint_ID = no
IO::recover = "autoprobe"
IO::out_proc_every = 2
IO::checkpoint_on_terminate = yes
IO::checkpoint_dir = "../checkpoints"
IO::recover_dir = "../checkpoints"
IO::abort_on_io_errors = yes
CarpetIOHDF5::open_one_input_file_at_a_time = yes
CarpetIOHDF5::compression_level = 0
################################################################################
# Run termination
################################################################################
TerminationTrigger::max_walltime = 24
# Trigger termination 30 minutes before the walltime is reached
TerminationTrigger::on_remaining_walltime = 30
TerminationTrigger::output_remtime_every_minutes = 30
TerminationTrigger::termination_from_file = yes
TerminationTrigger::termination_file = "terminate.txt"
TerminationTrigger::create_termination_file = yes
Cactus::terminate = time
Cactus::cctk_final_time = 3500.0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run.log
Type: text/x-log
Size: 202876 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20230116/d93d040c/attachment-0001.bin
More information about the Users
mailing list