<div dir="ltr"><div>Hi David</div><div><br></div><div>Changing the initial data (i.e. not using Kadath at all) from:<br></div><div>#ADMBase::metric_type                   = &quot;physical&quot;<br>#ADMBase::initial_data                   = &quot;Kadath&quot;<br>#ADMBase::initial_lapse                  = &quot;Kadath&quot;<br>#ADMBase::initial_shift                  = &quot;Kadath&quot;<br>#ADMBase::initial_dtlapse                = &quot;Kadath&quot;<br>#ADMBase::initial_dtshift                = &quot;Kadath&quot;<br>#HydroBase::initial_hydro                = &quot;Kadath&quot;<br># Kadath BHNS file<br>#KadathImporter::type = &quot;BHNS&quot;<br>#KadathImporter::filename = &quot;/home/topolski/Cactus/BHNS_par/BHNS.info&quot;<br>#KadathImporter::puncture_lapse = yes<br></div><div><br></div><div>to:<br>THC_InitialData::id_type              = &quot;shocktube&quot;<br>THC_InitialData::shocktube_case       = &quot;blast_wave&quot;<br>THC_InitialData::shocktube_dir        = &quot;x&quot;</div><div><br></div><div>did not change the result and the error is still the same. <br></div><div>In the ISSUES file of one of the THC thorns it says that Intel compilers do not pass all the tests - and I did compile with Intel - <br></div><div>but I think it&#39;s unlikely to cause this problem. <br></div><div><br></div><div>The single NS initial data and its parameter file supplied with Kadath works fine and I took the THC part from there. <br></div><div>This leads me to believe (as Erik points out too) I messed something up in grid structure / carpet parameters - I tried to follow the gallery example file for GW150914. <br></div><div><br></div><div>Best <br></div><div>Konrad <br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">pt., 13 sie 2021 o 17:38 Radice, David &lt;<a href="mailto:dur566@psu.edu">dur566@psu.edu</a>&gt; napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Konrad,<br>
<br>
I have never seen the code fail at that point. I suspect that your initial data thorn might be scheduling the BCs in a bin in which cctk_nghostzones is not available. Does your parfile work if you replace the NS-BH initial data with some trivial initial data from THC_InitialData?<br>
<br>
Best,<br>
<br>
David<br>
<br>
&gt; On Aug 13, 2021, at 4:57 AM, Konrad Topolski &lt;<a href="mailto:k.topolski2@student.uw.edu.pl" target="_blank">k.topolski2@student.uw.edu.pl</a>&gt; wrote:<br>
&gt; <br>
&gt; Hi all<br>
&gt; <br>
&gt; I&#39;m currently trying to run a BHNS simulation with initial data generated by Kadath elliptic solver (Fuka branch). <br>
&gt; <br>
&gt; After initial data import, I&#39;m getting an error from THC: <br>
&gt; WARNING level 0 from host nid00462 process 0<br>
&gt;   while executing schedule bin HydroBase_Select_Boundaries, routine THC_Core::THC_SelectBC<br>
&gt;   in thorn THC_Core, file /lustre/tetyda/home/topolski/Cactus/arrangements/THCCore/THC_Core/src/thc_select_bc.cc:76:<br>
&gt;   -&gt; Failed to select the BCs<br>
&gt; <br>
&gt; The simulation uses the following thorns NOT included with the official release of ET:<br>
&gt; <br>
&gt; HRSCCore<br>
&gt; Boost<br>
&gt; KadathImporter<br>
&gt; THCBase/CPPUtils, THCBase/FDCore ,THCBase/HRSCCore<br>
&gt; THCExtra/EOS_Barotropic, THCExtra/EOS_Thermal, THCExtra/EOS_Thermal_Idealgas<br>
&gt; <br>
&gt; Here&#39;s the parameter file. If I can provide more info, please let me know (also, if to upload it to a somewhat more convenient place).<br>
&gt; <br>
&gt; Best regards<br>
&gt; Konrad<br>
&gt; <br>
&gt; # The following is a modification of the ETK Gallery par file to utilize<br>
&gt; # Kadath initial data for a binary system of BH-NS<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Active thorns<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; ActiveThorns = &quot;<br>
&gt; ADMBase ML_ADMConstraints ADMCoupling ADMMacros<br>
&gt; AEILocalInterp AHFinderDirect Boundary Carpet CarpetIOASCII<br>
&gt; CarpetIOBasic CarpetIOHDF5 CarpetIOScalar CarpetInterp<br>
&gt; CarpetInterp2 CarpetLib CarpetReduce  CarpetRegrid2<br>
&gt; CarpetTracker CartGrid3D CoordBase Coordinates<br>
&gt; CoordinatesSymmetry GlobalDerivative hwloc<br>
&gt; IOUtil InitBase Interpolate2 QuasiLocalMeasures<br>
&gt; KadathImporter LocalInterp LoopControl MoL<br>
&gt; NaNChecker PunctureTracker Slab SpaceMask<br>
&gt; SphericalSurface StaticConformal SummationByParts<br>
&gt; SymBase SystemStatistics SystemTopology TerminationTrigger<br>
&gt; TensorTypes Time TmunuBase Vectors ML_BSSN<br>
&gt; ML_BSSN_Helper NewRad GenericFD WeylScal4<br>
&gt; Multipole ADMDerivatives CarpetSlab CoordGauge<br>
&gt; Dissipation EOS_Barotropic EOS_Thermal EOS_Thermal_Idealgas<br>
&gt; HydroBase HRSCCore ML_ADMConstraints THC_Core<br>
&gt; TimerReport BLAS  LAPACK  GSL Boost CPPUtils<br>
&gt; FDCore HDF5 MPI Timers CycleClock TGRtensor<br>
&gt; zlib Fortran VolumeIntegrals_GRMHD<br>
&gt; CarpetMask ReflectionSymmetry<br>
&gt; # currently turned off <br>
&gt; #GRHydro<br>
&gt; #PizzaBase<br>
&gt; #PizzaIDBase<br>
&gt; #PizzaNumUtils<br>
&gt; #NSTracker <br>
&gt; #Hydro_Analysis<br>
&gt; &quot;<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Grid structure<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; #------------------------------------------------------------------------------<br>
&gt; # Diagnostic parameters:<br>
&gt; #------------------------------------------------------------------------------<br>
&gt; Carpet::output_timers_every = 0<br>
&gt; Carpet::storage_verbose   = &quot;no&quot;<br>
&gt; Carpet::verbose           = &quot;no&quot;<br>
&gt; Carpet::veryverbose       = &quot;no&quot;<br>
&gt; Carpet::grid_structure_filename   = &quot;carpet-grid-structure&quot;<br>
&gt; Carpet::grid_coordinates_filename = &quot;carpet-grid-coordinates&quot;<br>
&gt; <br>
&gt; CarpetLib::output_bboxes  = &quot;no&quot;<br>
&gt; <br>
&gt; CarpetMask::verbose    = &quot;no&quot;<br>
&gt; CarpetReduce::verbose  = &quot;no&quot;<br>
&gt; CarpetRegrid2::verbose = &quot;no&quot;<br>
&gt; CarpetRegrid2::veryverbose    = &quot;no&quot;<br>
&gt; CarpetTracker::verbose = &quot;no&quot;<br>
&gt; <br>
&gt; <br>
&gt; TimerReport::out_every    = 4096<br>
&gt; TimerReport::out_filename = &quot;TimerReport&quot;<br>
&gt; TimerReport::output_all_timers          = &quot;yes&quot;<br>
&gt; TimerReport::output_all_timers_together = &quot;yes&quot;<br>
&gt; TimerReport::output_all_timers_readable = &quot;yes&quot;<br>
&gt; TimerReport::n_top_timers               = 40<br>
&gt; <br>
&gt; <br>
&gt; QuasiLocalMeasures::verbose   = &quot;no&quot;<br>
&gt; SphericalSurface::verbose   = &quot;no&quot;<br>
&gt; <br>
&gt; <br>
&gt; Carpet::domain_from_multipatch          = yes<br>
&gt; CartGrid3D::type                        = &quot;multipatch&quot;<br>
&gt; CartGrid3D::set_coordinate_ranges_on    = &quot;all maps&quot;<br>
&gt; Coordinates::coordinate_system          = &quot;Thornburg04&quot;<br>
&gt; Coordinates::h_cartesian                = 1.5574825174818182<br>
&gt; Coordinates::h_radial                   = 1.5574825174818182<br>
&gt; <br>
&gt; Coordinates::sphere_inner_radius        = 51.3969230769<br>
&gt; Coordinates::sphere_outer_radius        = 2194.15912088<br>
&gt; Coordinates::n_angular                  = 28<br>
&gt; <br>
&gt; Driver::ghost_size                      = 5<br>
&gt; Coordinates::patch_boundary_size        = 5<br>
&gt; Coordinates::additional_overlap_size    = 3<br>
&gt; Coordinates::outer_boundary_size        = 5<br>
&gt; <br>
&gt; CoordinatesSymmetry::reflection_z       = yes<br>
&gt; CoordinatesSymmetry::stagger            = no<br>
&gt; Coordinates::symmetry                   = &quot;+z bitant&quot;<br>
&gt; Coordinates::additional_symmetry_size   = 1<br>
&gt; Coordinates::verbose                    = no<br>
&gt; <br>
&gt; Time::timestep_method                   = &quot;given&quot;<br>
&gt; Time::timestep                          = 0.3504335664334091<br>
&gt; Carpet::time_refinement_factors         = &quot;[1,1,2,4,8,16,32,64,128,256]&quot;<br>
&gt; <br>
&gt; <br>
&gt; # General Carpet parameters:<br>
&gt; Carpet::enable_all_storage       = &quot;no&quot;<br>
&gt; Carpet::use_buffer_zones         = &quot;yes&quot;<br>
&gt; Carpet::schedule_barriers        = &quot;no&quot;<br>
&gt; <br>
&gt; Carpet::poison_new_timelevels    = &quot;yes&quot;<br>
&gt; Carpet::check_for_poison         = &quot;no&quot;<br>
&gt; <br>
&gt; Carpet::init_3_timelevels        = &quot;no&quot;<br>
&gt; Carpet::init_fill_timelevels     = &quot;yes&quot;<br>
&gt; <br>
&gt; CarpetLib::poison_new_memory         = &quot;yes&quot;<br>
&gt; CarpetLib::poison_value              = 114<br>
&gt; CarpetLib::check_bboxes              = &quot;no&quot;<br>
&gt; CarpetLib::interleave_communications = &quot;yes&quot;<br>
&gt; CarpetLib::combine_sends             = &quot;yes&quot;<br>
&gt; <br>
&gt; CarpetInterp::tree_search = &quot;yes&quot;<br>
&gt; CarpetInterp::check_tree_search = &quot;no&quot;<br>
&gt; <br>
&gt; CarpetRegrid2::freeze_unaligned_levels = &quot;yes&quot;<br>
&gt; CarpetRegrid2::freeze_unaligned_parent_levels = &quot;yes&quot;<br>
&gt; CarpetRegrid2::ensure_proper_nesting   = &quot;yes&quot;<br>
&gt; CarpetRegrid2::snap_to_coarse          = &quot;yes&quot;<br>
&gt; CarpetRegrid2::symmetry_rotating180    = &quot;no&quot;<br>
&gt; <br>
&gt; <br>
&gt; ###############################################################################<br>
&gt; # SpaceMask - only needed for GRHydro<br>
&gt; ###############################################################################<br>
&gt; <br>
&gt; #SpaceMask::use_mask = &quot;yes&quot;<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Mesh refinement<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; Carpet::max_refinement_levels           = 9<br>
&gt; CarpetRegrid2::num_centres              = 2<br>
&gt; CarpetRegrid2::num_levels_1             = 7<br>
&gt; CarpetRegrid2::position_x_1             = +17.37737<br>
&gt; CarpetRegrid2::radius_1                 = [0,21.267692,10.633846,5.316923,2.658462,1.329231,0.664615]<br>
&gt; CarpetRegrid2::num_levels_2             = 7<br>
&gt; CarpetRegrid2::position_x_2             = -13.76685<br>
&gt; CarpetRegrid2::radius_2                 = [0,17.132308,8.566154,4.283077,2.141538,1.070769,0.535385]<br>
&gt; <br>
&gt; Carpet::prolongation_order_space        = 5<br>
&gt; Carpet::prolongation_order_time         = 2<br>
&gt; <br>
&gt; CarpetRegrid2::regrid_every             = 128<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Initial data<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; HydroBase::timelevels        = 3<br>
&gt; HydroBase::prolongation_type =&quot;None&quot;<br>
&gt; ADMBase::metric_type = &quot;physical&quot;<br>
&gt; <br>
&gt; ADMBase::initial_data                   = &quot;Kadath&quot;<br>
&gt; ADMBase::initial_lapse                  = &quot;Kadath&quot;<br>
&gt; ADMBase::initial_shift                  = &quot;Kadath&quot;<br>
&gt; ADMBase::initial_dtlapse                = &quot;Kadath&quot;<br>
&gt; ADMBase::initial_dtshift                = &quot;Kadath&quot;<br>
&gt; HydroBase::initial_hydro                = &quot;Kadath&quot;<br>
&gt; <br>
&gt; # Kadath BNS file<br>
&gt; KadathImporter::type = &quot;BHNS&quot;<br>
&gt; #PizzaIDBase::eos_file = &quot;gam2.polytrope&quot;<br>
&gt; # Geometric unit system for initial data, specified by length unit.<br>
&gt; # use CACTUS units<br>
&gt; #PizzaBase::length_unit          = 1476.7161818921163<br>
&gt; KadathImporter::filename = &quot;/home/topolski/Cactus/BHNS_par/BHNS.info&quot;<br>
&gt; KadathImporter::puncture_lapse = yes<br>
&gt; <br>
&gt; InitBase::initial_data_setup_method     = &quot;init_single_level&quot;<br>
&gt; <br>
&gt; # ============================================================================<br>
&gt; # Templated hydrodynamics code<br>
&gt; # =============================================================================<br>
&gt; HydroBase::evolution_method             = &quot;THCode&quot;<br>
&gt; HydroBase::initial_entropy                                  = &quot;THCode&quot;<br>
&gt; <br>
&gt; THC_Core::physics                       = &quot;GRHD&quot;<br>
&gt; THC_Core::eos_type                      = &quot;ideal&quot;<br>
&gt; THC_Core::bc_type                       = &quot;none&quot;<br>
&gt; THC_Core::atmo_rho                      = 1e-13<br>
&gt; <br>
&gt; HRSCCore::scheme                        = &quot;FV&quot;<br>
&gt; HRSCCore::reconstruction                = &quot;MP5&quot;<br>
&gt; HRSCCore::riemann_solver                = &quot;HLLE&quot;<br>
&gt; <br>
&gt; THC_Core::c2a_BH_alp            = 0.15<br>
&gt; THC_Core::c2a_rho_strict                = 2.0e-5<br>
&gt; THC_Core::c2a_set_to_nan_on_failure             = &quot;no&quot;<br>
&gt; THC_Core::c2a_fix_conservatives         = &quot;yes&quot;<br>
&gt; THC_Core::c2a_kill_on_failure           = &quot;no&quot;<br>
&gt; <br>
&gt; EOS_Thermal::evol_eos_name = &quot;IdealGas&quot;<br>
&gt; EOS_Thermal_IdealGas::index_n = 1<br>
&gt; EOS_Thermal_IdealGas::eps_min = 1e-8<br>
&gt; <br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Evolution and boundary<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; GenericFD::jacobian_group = &quot;Coordinates::jacobian&quot;<br>
&gt; GenericFD::jacobian_derivative_group = &quot;Coordinates::jacobian2&quot;<br>
&gt; GenericFD::jacobian_identity_map = 0<br>
&gt; <br>
&gt; ADMBase::evolution_method         = &quot;ML_BSSN&quot;<br>
&gt; ADMBase::lapse_evolution_method   = &quot;ML_BSSN&quot;<br>
&gt; ADMBase::shift_evolution_method   = &quot;ML_BSSN&quot;<br>
&gt; ADMBase::dtlapse_evolution_method = &quot;ML_BSSN&quot;<br>
&gt; ADMBase::dtshift_evolution_method = &quot;ML_BSSN&quot;<br>
&gt; <br>
&gt; ADMBase::lapse_timelevels               = 3<br>
&gt; ADMBase::shift_timelevels               = 3<br>
&gt; ADMBase::metric_timelevels              = 3<br>
&gt; <br>
&gt; TmunuBase::prolongation_type            = &quot;none&quot;<br>
&gt; TmunuBase::stress_energy_storage                = &quot;yes&quot;<br>
&gt; TmunuBase::stress_energy_at_RHS         = &quot;yes&quot;<br>
&gt; TmunuBase::support_old_CalcTmunu_mechanism      = &quot;no&quot;<br>
&gt; <br>
&gt; <br>
&gt; ML_BSSN::timelevels                     = 3<br>
&gt; ML_BSSN::fdOrder             = 8<br>
&gt; THC_Core::fd_order                      = 4<br>
&gt; <br>
&gt; # 1+log slicing requires harmonicN = 1 and harmonicF = 2.0<br>
&gt; ML_BSSN::harmonicN           = 1<br>
&gt; ML_BSSN::harmonicF           = 2.0<br>
&gt; <br>
&gt; ML_BSSN::ShiftGammaCoeff     = 0.75<br>
&gt; ML_BSSN::evolveA                        = 1<br>
&gt; ML_BSSN::evolveB                        = 1<br>
&gt; ML_BSSN::AlphaDriver            = 0.0<br>
&gt; ML_BSSN::BetaDriver                     = 0.7    # ~ 1/M<br>
&gt; ML_BSSN::advectLapse         = 1<br>
&gt; ML_BSSN::advectShift         = 1<br>
&gt; <br>
&gt; ML_BSSN::MinimumLapse        = 1.0e-8<br>
&gt; <br>
&gt; # conformalaMethod = 1 for W, 0 for phi<br>
&gt; ML_BSSN::conformalMethod     = 1<br>
&gt; <br>
&gt; # We apply dissipation using GlobalDerivaitive so disable it here<br>
&gt; ML_BSSN::epsDiss             = 0.0<br>
&gt; <br>
&gt; ML_BSSN::dt_lapse_shift_method = &quot;noLapseShiftAdvection&quot;<br>
&gt; <br>
&gt; ML_BSSN::initial_boundary_condition = &quot;extrapolate-gammas&quot;<br>
&gt; ML_BSSN::rhs_boundary_condition     = &quot;NewRad&quot;<br>
&gt; Boundary::radpower                  = 2<br>
&gt; <br>
&gt; ML_BSSN::ML_log_confac_bound            = &quot;none&quot;<br>
&gt; ML_BSSN::ML_metric_bound                = &quot;none&quot;<br>
&gt; ML_BSSN::ML_Gamma_bound         = &quot;none&quot;<br>
&gt; ML_BSSN::ML_trace_curv_bound            = &quot;none&quot;<br>
&gt; ML_BSSN::ML_curv_bound          = &quot;none&quot;<br>
&gt; ML_BSSN::ML_lapse_bound         = &quot;none&quot;<br>
&gt; ML_BSSN::ML_dtlapse_bound               = &quot;none&quot;<br>
&gt; ML_BSSN::ML_shift_bound         = &quot;none&quot;<br>
&gt; ML_BSSN::ML_dtshift_bound               = &quot;none&quot;<br>
&gt; <br>
&gt; Dissipation::order                      = 5<br>
&gt; Dissipation::epsdis                     = 0.1<br>
&gt; Dissipation::vars                       = &quot;<br>
&gt; ML_BSSN::ML_log_confac<br>
&gt; ML_BSSN::ML_metric<br>
&gt; ML_BSSN::ML_curv<br>
&gt; ML_BSSN::ML_trace_curv<br>
&gt; ML_BSSN::ML_Gamma<br>
&gt; ML_BSSN::ML_lapse<br>
&gt; ML_BSSN::ML_shift<br>
&gt; &quot;<br>
&gt; <br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # BH and NS tracking (currently using VolumeIntegrals_GRMHD for tracking the <br>
&gt; # neutron star, will switch to NSTracker some time in the future) <br>
&gt; # BH is at x&gt;0, NS at x&lt;0<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; CarpetTracker::surface                      [0] = 0<br>
&gt; CarpetTracker::surface                      [1] = 1<br>
&gt; #PunctureTracker::track                      [0] = yes<br>
&gt; #PunctureTracker::initial_x                  [0] = -13.76685<br>
&gt; #PunctureTracker::which_surface_to_store_info[0] = 0<br>
&gt; PunctureTracker::track                      [0] = yes<br>
&gt; PunctureTracker::initial_x                  [0] = +17.37737<br>
&gt; PunctureTracker::which_surface_to_store_info[0] = 1<br>
&gt; <br>
&gt; <br>
&gt; VolumeIntegrals_GRMHD::verbose = 1<br>
&gt; VolumeIntegrals_GRMHD::VolIntegral_out_every = 32 <br>
&gt; VolumeIntegrals_GRMHD::NumIntegrals = 4<br>
&gt; # Integrate the entire volume with an integrand of 1. (used for testing/validation purposes only).<br>
&gt; VolumeIntegrals_GRMHD::Integration_quantity_keyword[1] = &quot;one&quot;<br>
&gt; # To compute the center of mass in an integration volume originally centered at (x,y,z) = (-13.76685,0,0) <br>
&gt; #with a coordinate radius of 13.5. Also use the center of mass integral result to set the ZEROTH AMR center.<br>
&gt; VolumeIntegrals_GRMHD::Integration_quantity_keyword[2] = &quot;centerofmass&quot; <br>
&gt; VolumeIntegrals_GRMHD::volintegral_sphere__center_x_initial         [2] = -13.76685<br>
&gt; VolumeIntegrals_GRMHD::volintegral_inside_sphere__radius            [2] =  9.10722<br>
&gt; VolumeIntegrals_GRMHD::amr_centre__tracks__volintegral_inside_sphere[2] =  0<br>
&gt; # Same as above, except use the integrand=1 (for validation purposes, to ensure the integration volume is approximately 4/3*pi*13.5^3).<br>
&gt; VolumeIntegrals_GRMHD::Integration_quantity_keyword[3] = &quot;one&quot;<br>
&gt; VolumeIntegrals_GRMHD::volintegral_sphere__center_x_initial         [3] = -13.76685<br>
&gt; VolumeIntegrals_GRMHD::volintegral_inside_sphere__radius            [3] =  9.10722<br>
&gt; # Perform rest-mass integrals over entire volume.<br>
&gt; VolumeIntegrals_GRMHD::Integration_quantity_keyword[4] = &quot;restmass&quot;<br>
&gt; VolumeIntegrals_GRMHD::enable_file_output = 1<br>
&gt; VolumeIntegrals_GRMHD::outVolIntegral_dir = &quot;volume_integration&quot;<br>
&gt; VolumeIntegrals_GRMHD::CoM_integrand_GAMMA_SPEED_LIMIT = 1e4<br>
&gt; <br>
&gt; <br>
&gt; ########################<br>
&gt; # Setting up NS Tracker and Hydro_Analysis<br>
&gt; #########################<br>
&gt; <br>
&gt; #Hydro_Analysis::Hydro_Analysis_comp_rho_max = &quot;true&quot;<br>
&gt; #Hydro_Analysis::Hydro_Analysis_rho_max_loc_only_positive_x = &quot;true&quot;<br>
&gt; #Hydro_Analysis::Hydro_Analysis_comp_rho_max_origin_distance = &quot;yes&quot;<br>
&gt; #Hydro_Analysis::Hydro_Analysis_average_multiple_maxima_locations = &quot;yes&quot;<br>
&gt; #Hydro_Analysis::Hydro_Analysis_interpolator_name = &quot;Lagrange polynomial interpolation (tensor product)&quot;<br>
&gt; <br>
&gt; #NSTracker::NSTracker_SF_Name          = &quot;Lefthand NS&quot;<br>
&gt; #NSTracker::NSTracker_SF_Index          = 0<br>
&gt; #NSTracker::NSTracker_max_distance = 3<br>
&gt; #NSTracker::NSTracker_tracked_location = &quot;Hydro_Analysis::Hydro_Analysis_rho_max_loc&quot; # location of maximum<br>
&gt; #position_x[NSTracker_SF_Index] = Hydro_Analysis_rho_max_loc<br>
&gt; #position_x[NSTracker_SF_Index_Opposite] = - Hydro_Analysis_rho_max_loc<br>
&gt; #NSTracker::NSTracker_tracked_location = &quot;Hydro_Analysis::Hydro_Analysis_rho_core_center_volume_weighted&quot; # center of mass<br>
&gt; #NSTracker::NSTracker_stop_time = -1<br>
&gt; <br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Spatial finite differencing<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; SummationByParts::order                              = 8<br>
&gt; <br>
&gt; # Drop order instead of using upwinded stencils, only for advection derivatives<br>
&gt; SummationByParts::sbp_upwind_deriv = no<br>
&gt; <br>
&gt; SummationByParts::sbp_1st_deriv                      = yes<br>
&gt; SummationByParts::sbp_2nd_deriv                      = no<br>
&gt; SummationByParts::onesided_interpatch_boundaries     = no<br>
&gt; SummationByParts::onesided_outer_boundaries          = yes<br>
&gt; SummationByParts::use_dissipation                    = no<br>
&gt; GlobalDerivative::use_dissipation                    = yes<br>
&gt; SummationByParts::scale_with_h                       = yes<br>
&gt; SummationByParts::dissipation_type                   = &quot;Kreiss-Oliger&quot;<br>
&gt; SummationByParts::epsdis                             = 0.15<br>
&gt; <br>
&gt; # Because dt/dx is half that of the other levels we also need half the dissipation<br>
&gt; GlobalDerivative::epsdis_for_level               [0] = 0.075<br>
&gt; <br>
&gt; # Variables for dissipation<br>
&gt; SummationByParts::vars                               = &quot;<br>
&gt; ML_BSSN::ML_log_confac<br>
&gt; ML_BSSN::ML_metric<br>
&gt; ML_BSSN::ML_trace_curv<br>
&gt; ML_BSSN::ML_curv<br>
&gt; ML_BSSN::ML_Gamma<br>
&gt; ML_BSSN::ML_lapse<br>
&gt; ML_BSSN::ML_shift<br>
&gt; ML_BSSN::ML_dtlapse<br>
&gt; ML_BSSN::ML_dtshift<br>
&gt; &quot;<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Time integration<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; MoL::ODE_Method                         = &quot;rk4&quot;<br>
&gt; MoL::MoL_Intermediate_Steps             = 4<br>
&gt; MoL::MoL_Num_Scratch_Levels             = 1<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Interpolation<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; # Use 5-th order interpatch interpolation on the Llama grid<br>
&gt; Interpolate::interpolator_order         = 5<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Psi4 computation<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; WeylScal4::fdOrder                   = 8<br>
&gt; WeylScal4::calc_scalars              = &quot;psis&quot;<br>
&gt; WeylScal4::calc_invariants           = &quot;always&quot;<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Psi4 mode decomposition<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; # Radii are chosen to be evenly spaced in 1/r as that is the variable<br>
&gt; # extrapolation is performed in<br>
&gt; Multipole::nradii       = 7<br>
&gt; Multipole::radius[0]    = 100<br>
&gt; Multipole::radius[1]    = 115<br>
&gt; Multipole::radius[2]    = 136<br>
&gt; Multipole::radius[3]    = 167<br>
&gt; Multipole::radius[4]    = 214<br>
&gt; Multipole::radius[5]    = 300<br>
&gt; Multipole::radius[6]    = 500<br>
&gt; Multipole::ntheta       = 120<br>
&gt; Multipole::nphi         = 240<br>
&gt; Multipole::variables    = &quot;WeylScal4::Psi4r{sw=-2 cmplx=&#39;WeylScal4::Psi4i&#39; name=&#39;psi4&#39;}&quot;<br>
&gt; Multipole::out_every    = 256<br>
&gt; Multipole::l_max        = 8<br>
&gt; Multipole::output_hdf5  = yes<br>
&gt; <br>
&gt; # Disable ASCII output to avoid creating a large number of files<br>
&gt; Multipole::output_ascii = no<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Apparent Horizons<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; AHFinderDirect::N_horizons                               = 2<br>
&gt; AHFinderDirect::find_every                               = 256<br>
&gt; AHFinderDirect::output_h_every                           = 0<br>
&gt; AHFinderDirect::max_Newton_iterations__initial           = 50<br>
&gt; AHFinderDirect::max_Newton_iterations__subsequent        = 50<br>
&gt; AHFinderDirect::max_allowable_Theta_growth_iterations    = 10<br>
&gt; AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10<br>
&gt; AHFinderDirect::geometry_interpolator_name               = &quot;Lagrange polynomial interpolation&quot;<br>
&gt; AHFinderDirect::geometry_interpolator_pars               = &quot;order=4&quot;<br>
&gt; AHFinderDirect::surface_interpolator_name                = &quot;Lagrange polynomial interpolation&quot;<br>
&gt; AHFinderDirect::surface_interpolator_pars                = &quot;order=4&quot;<br>
&gt; AHFinderDirect::verbose_level                            = &quot;physics details&quot;<br>
&gt; AHFinderDirect::move_origins                             = yes<br>
&gt; <br>
&gt; <br>
&gt; AHFinderDirect::origin_x                             [1] = +17.37737<br>
&gt; AHFinderDirect::initial_guess__coord_sphere__x_center[1] = +17.37737<br>
&gt; AHFinderDirect::initial_guess__coord_sphere__radius  [1] = 0.73<br>
&gt; AHFinderDirect::which_surface_to_store_info          [1] = 2<br>
&gt; AHFinderDirect::set_mask_for_individual_horizon      [1] = no<br>
&gt; AHFinderDirect::reset_horizon_after_not_finding      [1] = no<br>
&gt; AHFinderDirect::track_origin_from_grid_scalar        [1] = yes<br>
&gt; AHFinderDirect::track_origin_source_x                [1] = &quot;PunctureTracker::pt_loc_x[0]&quot;<br>
&gt; AHFinderDirect::track_origin_source_y                [1] = &quot;PunctureTracker::pt_loc_y[0]&quot;<br>
&gt; AHFinderDirect::track_origin_source_z                [1] = &quot;PunctureTracker::pt_loc_z[0]&quot;<br>
&gt; AHFinderDirect::max_allowable_horizon_radius         [1] = 3<br>
&gt; <br>
&gt; #AHFinderDirect::origin_x                             [2] = -5.53846153846<br>
&gt; #AHFinderDirect::initial_guess__coord_sphere__x_center[2] = -5.53846153846<br>
&gt; #AHFinderDirect::initial_guess__coord_sphere__radius  [2] = 0.535384615385<br>
&gt; #AHFinderDirect::which_surface_to_store_info          [2] = 3<br>
&gt; #AHFinderDirect::set_mask_for_individual_horizon      [2] = no<br>
&gt; #AHFinderDirect::reset_horizon_after_not_finding      [2] = no<br>
&gt; #AHFinderDirect::track_origin_from_grid_scalar        [2] = yes<br>
&gt; #AHFinderDirect::track_origin_source_x                [2] = &quot;PunctureTracker::pt_loc_x[1]&quot;<br>
&gt; #AHFinderDirect::track_origin_source_y                [2] = &quot;PunctureTracker::pt_loc_y[1]&quot;<br>
&gt; #AHFinderDirect::track_origin_source_z                [2] = &quot;PunctureTracker::pt_loc_z[1]&quot;<br>
&gt; #AHFinderDirect::max_allowable_horizon_radius         [2] = 3<br>
&gt; <br>
&gt; AHFinderDirect::origin_x                             [2] = 0<br>
&gt; AHFinderDirect::find_after_individual                [2] = 371902<br>
&gt; AHFinderDirect::initial_guess__coord_sphere__x_center[2] = 0<br>
&gt; AHFinderDirect::initial_guess__coord_sphere__radius  [2] = 1.67<br>
&gt; AHFinderDirect::which_surface_to_store_info          [2] = 4<br>
&gt; AHFinderDirect::set_mask_for_individual_horizon      [2] = no<br>
&gt; AHFinderDirect::max_allowable_horizon_radius         [2] = 6<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Spherical surfaces<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; SphericalSurface::nsurfaces             = 4<br>
&gt; SphericalSurface::maxntheta             = 66<br>
&gt; SphericalSurface::maxnphi               = 124<br>
&gt; <br>
&gt; # Surfaces 0 and 1 are used by PunctureTracker<br>
&gt; <br>
&gt; # Horizon 1<br>
&gt; SphericalSurface::ntheta            [2] = 41<br>
&gt; SphericalSurface::nphi              [2] = 80<br>
&gt; SphericalSurface::nghoststheta      [2] = 2<br>
&gt; SphericalSurface::nghostsphi        [2] = 2<br>
&gt; <br>
&gt; # Horizon 2<br>
&gt; #SphericalSurface::ntheta            [3] = 41<br>
&gt; #SphericalSurface::nphi              [3] = 80<br>
&gt; #SphericalSurface::nghoststheta      [3] = 2<br>
&gt; #SphericalSurface::nghostsphi        [3] = 2<br>
&gt; <br>
&gt; # Common horizon<br>
&gt; SphericalSurface::ntheta            [3] = 41<br>
&gt; SphericalSurface::nphi              [3] = 80<br>
&gt; SphericalSurface::nghoststheta      [3] = 2<br>
&gt; SphericalSurface::nghostsphi        [3] = 2<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Isolated Horizons<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; QuasiLocalMeasures::veryverbose            = no<br>
&gt; QuasiLocalMeasures::interpolator           = &quot;Lagrange polynomial interpolation&quot;<br>
&gt; QuasiLocalMeasures::interpolator_options   = &quot;order=4&quot;<br>
&gt; QuasiLocalMeasures::spatial_order          = 4<br>
&gt; QuasiLocalMeasures::num_surfaces           = 3<br>
&gt; QuasiLocalMeasures::surface_index      [0] = 2<br>
&gt; QuasiLocalMeasures::surface_index      [1] = 3<br>
&gt; #QuasiLocalMeasures::surface_index      [2] = 4<br>
&gt; QuasiLocalMeasures::output_vtk_every       = 0<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Correctness checking<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; <br>
&gt; NaNChecker::check_every                 = 256<br>
&gt; NanChecker::check_after                 = 0<br>
&gt; NaNChecker::report_max                  = 10<br>
&gt; NaNChecker::verbose                     = &quot;all&quot;<br>
&gt; NaNChecker::action_if_found             = terminate<br>
&gt; NaNChecker::out_NaNmask                 = yes<br>
&gt; NaNChecker::check_vars                  = &quot;<br>
&gt; ML_BSSN::ML_log_confac<br>
&gt; HydroBase::w_lorentz<br>
&gt; ADMBase::curv <br>
&gt; ADMBase::metric <br>
&gt; ADMBase::lapse <br>
&gt; ADMBase::shift <br>
&gt; HydroBase::rho <br>
&gt; HydroBase::eps <br>
&gt; HydroBase::press <br>
&gt; HydroBase::vel<br>
&gt; &quot;<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Timers<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; Carpet::output_timer_tree_every         = 1024<br>
&gt; Carpet::output_initialise_timer_tree    = yes<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Output<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; IO::out_dir                             = &quot;data&quot;<br>
&gt; CarpetIOScalar::all_reductions_in_one_file    = &quot;no&quot;<br>
&gt; CarpetIOScalar::one_file_per_group            = &quot;no&quot;<br>
&gt; CarpetIOASCII::one_file_per_group             = &quot;no&quot;<br>
&gt; CarpetIOHDF5::one_file_per_group              = &quot;no&quot;<br>
&gt; <br>
&gt; <br>
&gt; IOBasic::outInfo_every                  = 1<br>
&gt; IOBasic::outInfo_reductions             = &quot;minimum maximum&quot;<br>
&gt; IOBasic::outInfo_vars                   = &quot;<br>
&gt; ML_BSSN::ML_Ham<br>
&gt; ML_BSSN::ML_log_confac<br>
&gt; Carpet::physical_time_per_hour<br>
&gt; SystemStatistics::maxrss_mb<br>
&gt; SystemStatistics::swap_used_mb<br>
&gt; Carpet::physical_time_per_hour<br>
&gt; HydroBase::rho<br>
&gt; #GRHydro::dens{reductions = &#39;sum maximum&#39;}<br>
&gt; HydroBase::w_lorentz<br>
&gt; &quot;<br>
&gt; <br>
&gt; IOScalar::outScalar_every               = 256<br>
&gt; IOScalar::outScalar_reductions          = &quot;minimum maximum average&quot;<br>
&gt; IOScalar::outScalar_vars                = &quot;<br>
&gt; SystemStatistics::process_memory_mb<br>
&gt; ML_BSSN::ML_Ham{ reductions = &#39;maximum&#39; }<br>
&gt; &quot;<br>
&gt; <br>
&gt; IOASCII::out0D_every                    = 256<br>
&gt; IOASCII::out0D_vars                     = &quot;<br>
&gt; Carpet::physical_time_per_hour<br>
&gt; Carpet::timing<br>
&gt; PunctureTracker::pt_loc<br>
&gt; QuasiLocalMeasures::qlm_scalars{out_every = 256}<br>
&gt; &quot;<br>
&gt; <br>
&gt; IOASCII::out1D_every                    = 0<br>
&gt; IOASCII::out1D_d                        = no<br>
&gt; IOASCII::out1D_x                        = yes<br>
&gt; IOASCII::out1D_y                        = no<br>
&gt; IOASCII::out1D_z                        = yes<br>
&gt; IOASCII::out1D_vars                     = &quot;<br>
&gt; ML_BSSN::ML_log_confac<br>
&gt; ML_BSSN::ML_trace_curv<br>
&gt; WeylScal4::Psi4r<br>
&gt; &quot;<br>
&gt; <br>
&gt; IOASCII::out2D_every                    = 0<br>
&gt; IOASCII::out2D_vars                     = &quot;<br>
&gt; &quot;<br>
&gt; <br>
&gt; IOHDF5::out_every                       = 0<br>
&gt; IOHDF5::out_vars                        = &quot;<br>
&gt; Grid::Coordinates{out_every=1000000000 refinement_levels={0}}<br>
&gt; ML_BSSN::ML_log_confac<br>
&gt; WeylScal4::Psi4r<br>
&gt; WeylScal4::Psi4i<br>
&gt; WeylScal4::curvIr{refinement_levels={3 5}}<br>
&gt; WeylScal4::curvIi{refinement_levels={3 5}}<br>
&gt; WeylScal4::curvJr{refinement_levels={3 5}}<br>
&gt; WeylScal4::curvJi{refinement_levels={3 5}}<br>
&gt; &quot;<br>
&gt; <br>
&gt; IOHDF5::out2D_every                     = 64 <br>
&gt; IOHDF5::out2D_vars                      = &quot;<br>
&gt; Grid::Coordinates{out_every=1000000000 refinement_levels={0}}<br>
&gt; ML_BSSN::ML_Ham<br>
&gt; ML_BSSN::alpha<br>
&gt; ML_BSSN::phi<br>
&gt; ML_BSSN::trK<br>
&gt; WeylScal4::Psi4r<br>
&gt; WeylScal4::Psi4i<br>
&gt; HydroBase::rho<br>
&gt; HydroBase::press<br>
&gt; HydroBase::w_lorentz<br>
&gt; &quot;<br>
&gt; <br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Checkpointing and recovery<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; #CarpetIOHDF5::checkpoint                    = yes<br>
&gt; IOHDF5::checkpoint                          = &quot;yes&quot;<br>
&gt; IO::checkpoint_ID                           = &quot;yes&quot;<br>
&gt; IO::recover                                 = &quot;autoprobe&quot;<br>
&gt; IO::checkpoint_every_walltime_hours = 1.5<br>
&gt; IO::checkpoint_keep=2<br>
&gt; IO::checkpoint_on_terminate         = &quot;yes&quot;<br>
&gt; IO::out_mode                                = &quot;onefile&quot;<br>
&gt; IO::checkpoint_dir                          = &quot;../checkpoints&quot;<br>
&gt; IO::recover_dir                             = &quot;../checkpoints&quot;<br>
&gt; IO::abort_on_io_errors                      = yes<br>
&gt; CarpetIOHDF5::open_one_input_file_at_a_time = yes<br>
&gt; CarpetIOHDF5::compression_level             = 0<br>
&gt; <br>
&gt; ################################################################################<br>
&gt; # Run termination<br>
&gt; ################################################################################<br>
&gt; <br>
&gt; TerminationTrigger::max_walltime                 = 48<br>
&gt; # Trigger termination 30 minutes before the walltime is reached<br>
&gt; TerminationTrigger::on_remaining_walltime        = 30<br>
&gt; TerminationTrigger::output_remtime_every_minutes = 30<br>
&gt; TerminationTrigger::termination_from_file        = yes<br>
&gt; TerminationTrigger::termination_file             = &quot;terminate.txt&quot;<br>
&gt; TerminationTrigger::create_termination_file      = yes<br>
&gt; <br>
&gt; Cactus::terminate                               = time<br>
&gt; Cactus::cctk_final_time                         = 1900.0<br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; _______________________________________________<br>
&gt; Users mailing list<br>
&gt; <a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
&gt; <a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
</blockquote></div>