[Users] Loss of convergence with subcycling in time

Miguel Zilhão mzilhao at ua.pt
Wed Nov 19 05:17:45 CST 2025


hi all,

we've recently added the option to compute the Gauss constraint in the 
ProcaEvolve thorn 
(https://bitbucket.org/canuda/proca/src/experimental_miguel/ProcaEvolve/) 
and we were surprised to see that, even for very small evolutions of a 
simple (unperturbed) charged black hole, this constraint violation does 
not converge. of course everything is converging just fine at t=0, but 
once the evolution starts, convergence is lost really quickly.

after experimenting with different things, i think i've narrowed the 
issue down to the subcycling in time. i've attached some plots as well 
as the corresponding parameter files. these plots are not convergence 
tests, it's just what i'm observing with and without subcycling in time 
(everything else is the same as you can see from the parfiles).

these are evolutions for a single charged BH, with Q=0.2.
in one of the parfiles (LeanBSSN_RN_Q0.2_nosubcycl_hf40_202509.par) all 
timelevels update at the same rate with

   Carpet::time_refinement_factors         = "[1, 1, 1, 1, 1, 1, 1]"

for the other parfile (LeanBSSN_RN_Q0.2_hf40_202509.par), all timelevels 
update at the same rate *except* the two inner ones. for the "base" grid 
functions, i see no noticeable difference insofar as i've checked, but 
for the Gauss constraint violation the results are very different as you 
can see from the plots attached:

- in the gc_x_RN_nosubcycl.pdf plot, everything looks fine
- in the gc_x_RN_subcycl.pdf case, notice how a lot of noise propagates 
from the two inner refinements levels -- the ones that updated more 
frequently than the rest.

the problem is that this noise is not contained in the buffer region (if 
it was, we'd be fine), but propagates out, contaminates the rest of the 
grid, and totally ruins any convergence study.

actually a similar thing also happens for the hamiltonian constraint -- 
plots also attached (hc_x_RN_nosubcycl.pdf, hc_x_RN_subcycl.pdf) -- but 
in this case the effect is smaller and it doesn't spoil convergence studies.

so i was wondering whether this is at all expected and/or if there are 
some parameters or setting that we may have overlooked?

thanks,
Miguel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hc_x_RN_nosubcycl.pdf
Type: application/pdf
Size: 12898 bytes
Desc: hc_x_RN_nosubcycl.pdf
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20251119/2edf785c/attachment-0004.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hc_x_RN_subcycl.pdf
Type: application/pdf
Size: 13004 bytes
Desc: hc_x_RN_subcycl.pdf
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20251119/2edf785c/attachment-0005.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gc_x_RN_subcycl.pdf
Type: application/pdf
Size: 13022 bytes
Desc: gc_x_RN_subcycl.pdf
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20251119/2edf785c/attachment-0006.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gc_x_RN_nosubcycl.pdf
Type: application/pdf
Size: 12863 bytes
Desc: gc_x_RN_nosubcycl.pdf
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20251119/2edf785c/attachment-0007.pdf>
-------------- next part --------------

#------------------------------------------------------------------------------
ActiveThorns = "
  ADMBase
  ADMCoupling
  ADMMacros
  AEILocalInterp
  AHFinderDirect
  Boundary
  Carpet
  CarpetInterp
  CarpetIOASCII
  CarpetIOBasic
  CarpetIOHDF5
  CarpetIOScalar
  CarpetLib
  CarpetMask
  CarpetReduce
  CarpetRegrid2
  CarpetSlab
  CarpetTracker
  CartGrid3D
  CoordBase
  CoordGauge
  Dissipation
  # Formaline
  Fortran
  GenericFD
  GSL
  HDF5
  InitBase
  IOUtil
  LeanBSSNMoL
  LocalInterp
  LoopControl
  # ML_ADMConstraints
  # ML_BSSN
  # ML_BSSN_Helper
  MoL
  Multipole
  NaNChecker
  NewRad
#  NPScalars_Proca
  ProcaBase
#  Proca_simpleID
  Proca_EMG_ID
  ProcaEvolve
 # PunctureTracker
  QuasiLocalMeasures
  ReflectionSymmetry
 # RotatingSymmetry180
  Slab
  SpaceMask
  SphericalSurface
  StaticConformal
  # SummationByParts
  SymBase
  SystemStatistics
  TerminationTrigger
  Time
  TimerReport
  TmunuBase
  # TwoPunctures
  # TwoPunctures_KerrProca
  # Vectors
  # WeylScal4
"
#------------------------------------------------------------------------------


# Grid setup
#------------------------------------------------------------------------------

CartGrid3D::type                     = "coordbase"
Carpet::domain_from_coordbase        = yes
CoordBase::domainsize                = "minmax"

# make sure all (xmax - xmin)/dx are integers!
CoordBase::xmin                      = -128.00
CoordBase::ymin                      = -128.00
CoordBase::zmin                      =    0.00
CoordBase::xmax                      = +128.00
CoordBase::ymax                      = +128.00
CoordBase::zmax                      = +128.00
CoordBase::dx                        =    1.6
CoordBase::dy                        =    1.6
CoordBase::dz                        =    1.6

driver::ghost_size                   = 3

CoordBase::boundary_size_x_lower     = 3
CoordBase::boundary_size_y_lower     = 3
CoordBase::boundary_size_z_lower     = 3
CoordBase::boundary_size_x_upper     = 3
CoordBase::boundary_size_y_upper     = 3
CoordBase::boundary_size_z_upper     = 3

CoordBase::boundary_shiftout_x_lower = 0
CoordBase::boundary_shiftout_y_lower = 0
CoordBase::boundary_shiftout_z_lower = 1

CarpetRegrid2::symmetry_rotating180  = no

ReflectionSymmetry::reflection_x     = no
ReflectionSymmetry::reflection_y     = no
ReflectionSymmetry::reflection_z     = yes
ReflectionSymmetry::avoid_origin_x   = yes
ReflectionSymmetry::avoid_origin_y   = yes
ReflectionSymmetry::avoid_origin_z   = no


# Mesh refinement
#------------------------------------------------------------------------------

Carpet::max_refinement_levels           = 7

CarpetRegrid2::num_centres              = 1

CarpetRegrid2::num_levels_1             = 7
CarpetRegrid2::radius_1[1]              = 64.0
CarpetRegrid2::radius_1[2]              = 32.0
CarpetRegrid2::radius_1[3]              = 16.0
CarpetRegrid2::radius_1[4]              =  8.0
CarpetRegrid2::radius_1[5]              =  4.0
CarpetRegrid2::radius_1[6]              =  2.0
#CarpetRegrid2::radius_1[7]              =  1.0
#CarpetRegrid2::radius_1[8]              =  0.75


Carpet::use_buffer_zones                = yes
Carpet::prolongation_order_space        = 5
Carpet::prolongation_order_time         = 2

CarpetRegrid2::freeze_unaligned_levels  = yes
CarpetRegrid2::regrid_every             = 0

CarpetRegrid2::verbose                  = yes

Carpet::grid_structure_filename         = "carpet-grid-structure"
Carpet::grid_coordinates_filename       = "carpet-grid-coordinates"

Carpet::time_refinement_factors         = "[1, 1, 1, 1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512]"
Time::dtfac                             = 0.03125


# Initial Data
#------------------------------------------------------------------------------

ProcaBase::mu                         = 0.0

ADMBase::initial_data                 = "Proca_EMG_RN"
ADMBase::initial_lapse                = "psi^n"
ADMBase::initial_shift                = "zero"
ADMBase::initial_dtlapse              = "zero"
ADMBase::initial_dtshift              = "zero"

ADMBase::lapse_timelevels             = 3
ADMBase::shift_timelevels             = 3
ADMBase::metric_timelevels            = 3

Proca_EMG_ID::initial_lapse_psi_exponent  = -2.0

Proca_EMG_ID::initial_Aphi            = "zero"

Proca_EMG_ID::M                       = 1.0
Proca_EMG_ID::Q                       = 0.2
Proca_EMG_ID::x0                      = 1.0d-5


InitBase::initial_data_setup_method   = "init_some_levels"
Carpet::init_fill_timelevels          = yes
Carpet::init_3_timelevels             = no


# Evolution
#------------------------------------------------------------------------------

TmunuBase::stress_energy_storage  = yes
TmunuBase::stress_energy_at_RHS   = yes
TmunuBase::timelevels             = 1
TmunuBase::prolongation_type      = "none"

ProcaBase::evolution_method       = "ProcaEvolve"

ADMBase::evolution_method               = "LeanBSSNMoL"
ADMBase::lapse_evolution_method         = "LeanBSSNMoL"
ADMBase::shift_evolution_method         = "LeanBSSNMoL"
ADMBase::dtlapse_evolution_method       = "LeanBSSNMoL"
ADMBase::dtshift_evolution_method       = "LeanBSSNMoL"

LeanBSSNMoL::impose_conf_fac_floor_at_initial  = yes
LeanBSSNMoL::eta_beta                     = 1
LeanBSSNMoL::eta_transition               = no
#LeanBSSNMoL::eta_transition_r             = 60
LeanBSSNMoL::conf_fac_floor               = 1.0d-04
LeanBSSNMoL::derivs_order                 = 4
LeanBSSNMoL::use_advection_stencils       = yes
LeanBSSNMoL::calculate_constraints        = yes

ProcaEvolve::calculate_constraints        = yes
ProcaEvolve::Zeta_BC_type                 = "scalar"


# Spatial finite differencing
#------------------------------------------------------------------------------

Dissipation::epsdis = 0.2
Dissipation::order  = 5
Dissipation::vars   = "
  ADMBase::lapse
  ADMBase::shift
  LeanBSSNMoL::conf_fac
  LeanBSSNMoL::hmetric
  LeanBSSNMoL::hcurv
  LeanBSSNMoL::trk
  LeanBSSNMoL::gammat
  ProcaBase::Ei
  ProcaBase::Ai
  ProcaBase::Aphi
  ProcaBase::Zeta
"


# Integration method
#------------------------------------------------------------------------------

MoL::ODE_Method                 = "RK4"
MoL::MoL_Intermediate_Steps     = 4
MoL::MoL_Num_Scratch_Levels     = 1

Carpet::num_integrator_substeps = 4


# Spherical surfaces
#------------------------------------------------------------------------------

SphericalSurface::nsurfaces = 1
SphericalSurface::maxntheta = 66
SphericalSurface::maxnphi   = 124
SphericalSurface::verbose   = no

# Horizon 1
SphericalSurface::ntheta            [0] = 41
SphericalSurface::nphi              [0] = 80
SphericalSurface::nghoststheta      [0] = 2
SphericalSurface::nghostsphi        [0] = 2
CarpetMask::excluded_surface        [0] = 0
CarpetMask::excluded_surface_factor [0] = 1.0

CarpetMask::verbose = no



# # Wave extraction
# #------------------------------------------------------------------------------

# Multipole::nradii       = 3
# Multipole::radius[0]    = 30
# Multipole::radius[1]    = 80
# Multipole::radius[2]    = 100
# Multipole::ntheta       = 120
# Multipole::nphi         = 240
# Multipole::variables    = "
#   NPScalars::psi4re{sw=-2 cmplx='NPScalars::psi4im' name='Psi4'}
# "
# Multipole::integration_method = "Simpson"

# Multipole::out_every    = 128
# Multipole::l_max        = 4
# Multipole::output_hdf5  = no
# Multipole::output_ascii = yes


# Horizons
#------------------------------------------------------------------------------

# AHFinderDirect::verbose_level                           = "algorithm highlights"
AHFinderDirect::verbose_level                            = "physics details"
AHFinderDirect::output_BH_diagnostics                    = "true"
AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES       = no

AHFinderDirect::N_horizons                               = 1
AHFinderDirect::find_every                               = 128

AHFinderDirect::output_h_every                           = 0
AHFinderDirect::max_Newton_iterations__initial           = 50
AHFinderDirect::max_Newton_iterations__subsequent        = 50
AHFinderDirect::max_allowable_Theta_growth_iterations    = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name               = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars               = "order=4"
AHFinderDirect::surface_interpolator_name                = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars                = "order=4"

AHFinderDirect::move_origins                             = no

AHFinderDirect::origin_x                             [1] = 0
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = 0
AHFinderDirect::initial_guess__coord_sphere__radius  [1] = 0.3
AHFinderDirect::which_surface_to_store_info          [1] = 0
AHFinderDirect::set_mask_for_individual_horizon      [1] = no
AHFinderDirect::reset_horizon_after_not_finding      [1] = no
AHFinderDirect::max_allowable_horizon_radius         [1] = 5
#AHFinderDirect::dont_find_after_individual_time      [1] = 30.0


# Isolated Horizons
#-------------------------------------------------------------------------------

QuasiLocalMeasures::verbose                = yes
QuasiLocalMeasures::veryverbose            = no
QuasiLocalMeasures::interpolator           = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options   = "order=4"
QuasiLocalMeasures::spatial_order          = 4
QuasiLocalMeasures::num_surfaces           = 1
QuasiLocalMeasures::surface_index      [0] = 0


# Check for NaNs
#-------------------------------------------------------------------------------

Carpet::poison_new_timelevels = yes
CarpetLib::poison_new_memory  = yes
Carpet::check_for_poison      = no

NaNChecker::check_every     = 512
NanChecker::check_after     = 0
NaNChecker::report_max      = 10
# NaNChecker::verbose         = "all"
NaNChecker::action_if_found = "terminate"
NaNChecker::out_NaNmask     = yes
NaNChecker::check_vars      = "
  LeanBSSNMoL::conf_fac
"


# Timers
#-------------------------------------------------------------------------------

Cactus::cctk_timer_output               = "full"
TimerReport::out_every                  = 5120
TimerReport::n_top_timers               = 40
TimerReport::output_all_timers_together = yes
TimerReport::output_all_timers_readable = yes
TimerReport::output_schedule_timers     = no


# I/O thorns
#-------------------------------------------------------------------------------

Cactus::cctk_run_title       = $parfile
IO::out_dir                  = $parfile

IOScalar::one_file_per_group = yes
IOASCII::one_file_per_group  = yes

IOHDF5::use_checksums        = no
IOHDF5::one_file_per_group   = no

IOBasic::outInfo_every       = 1
IOBasic::outInfo_reductions  = "minimum maximum"
IOBasic::outInfo_vars        = "
  Carpet::physical_time_per_hour
  LeanBSSNMoL::conf_fac
  SystemStatistics::maxrss_mb
"

# for scalar reductions of 3D grid functions
IOScalar::outScalar_every               = 4
IOScalar::outScalar_reductions          = "minimum maximum average norm2"
IOScalar::outScalar_vars                = "
  ADMBase::lapse
  LeanBSSNMoL::ham
  LeanBSSNMoL::mom
  ProcaBase::Zeta
  ProcaEvolve::gc
"

# # output just at one point (0D)
# IOASCII::out0D_every    = 64
# IOASCII::out0D_point_x  = 4.0
# IOASCII::out0D_point_y  = 4.0
# IOASCII::out0D_vars  = "
#   ADMBase::lapse
#   ADMBase::shift
#   Carpet::timing
#   QuasiLocalMeasures::qlm_scalars{out_every = 256}
# "

# IOASCII::output_symmetry_points = no
# IOASCII::out3D_ghosts           = no

# 1D text output
IOASCII::out1D_every            = 4
IOASCII::out1D_d                = no
IOASCII::out1D_x                = yes
IOASCII::out1D_y                = no
IOASCII::out1D_z                = yes
IOASCII::out1D_vars             = "
  ADMBase::lapse
  ADMBase::shift
  ProcaBase::Ei
  ProcaBase::Ai
  ProcaBase::Aphi
  ProcaBase::Zeta
  ProcaEvolve::gc
#  LeanBSSNMoL::hmetric
#  LeanBSSNMoL::trk
  LeanBSSNMoL::conf_fac
  LeanBSSNMoL::ham
  LeanBSSNMoL::mom
"

# 1D HDF5 output
#IOHDF5::out1D_every            = 256
#IOHDF5::out1D_d                = no
#IOHDF5::out1D_x                = yes
#IOHDF5::out1D_y                = no
#IOHDF5::out1D_z                = no
#IOHDF5::out1D_vars             = "
#  ADMBase::lapse
#"

# # 2D HDF5 output
# IOHDF5::out2D_every             = 512
# IOHDF5::out2D_xy                = yes
# IOHDF5::out2D_xz                = no
# IOHDF5::out2D_yz                = yes
# IOHDF5::out2D_vars              = "
#   ADMBase::lapse
#   ProcaBase::Aphi
#   LeanBSSNMoL::ham
# #  LeanBSSNMoL::mom
#   LeanBSSNMoL::conf_fac
# "

# # 3D HDF5 output
# IOHDF5::out_every                      = 8192
# IOHDF5::out_vars                       = "
#   ADMBase::lapse
# "

Carpet::verbose                    = no
Carpet::veryverbose                = no
Carpet::schedule_barriers          = no
Carpet::storage_verbose            = no
CarpetLib::output_bboxes           = no

Cactus::cctk_full_warnings         = yes
Cactus::highlight_warning_messages = no


# Checkpointing and recovery
#-------------------------------------------------------------------------------

CarpetIOHDF5::checkpoint             = yes
IO::checkpoint_dir                   = "../checkpoints"
IO::checkpoint_ID                    = no
IO::checkpoint_every_walltime_hours  = 23
IO::checkpoint_on_terminate          = yes
IO::out_proc_every                   = 2
IO::checkpoint_keep                  = 1

IO::recover                          = "autoprobe"
IO::recover_dir                      = "../checkpoints"

IO::abort_on_io_errors                      = yes
CarpetIOHDF5::open_one_input_file_at_a_time = yes
CarpetIOHDF5::compression_level             = 9


# Run termination
#-------------------------------------------------------------------------------

TerminationTrigger::max_walltime                 = 72 # hours
TerminationTrigger::on_remaining_walltime        = 60 # minutes
TerminationTrigger::output_remtime_every_minutes = 60
TerminationTrigger::termination_from_file        = yes
TerminationTrigger::termination_file             = "terminate.txt"
TerminationTrigger::create_termination_file      = yes

Cactus::terminate       = "time"
Cactus::cctk_final_time = 100.0
-------------- next part --------------

#------------------------------------------------------------------------------
ActiveThorns = "
  ADMBase
  ADMCoupling
  ADMMacros
  AEILocalInterp
  AHFinderDirect
  Boundary
  Carpet
  CarpetInterp
  CarpetIOASCII
  CarpetIOBasic
  CarpetIOHDF5
  CarpetIOScalar
  CarpetLib
  CarpetMask
  CarpetReduce
  CarpetRegrid2
  CarpetSlab
  CarpetTracker
  CartGrid3D
  CoordBase
  CoordGauge
  Dissipation
  # Formaline
  Fortran
  GenericFD
  GSL
  HDF5
  InitBase
  IOUtil
  LeanBSSNMoL
  LocalInterp
  LoopControl
  # ML_ADMConstraints
  # ML_BSSN
  # ML_BSSN_Helper
  MoL
  Multipole
  NaNChecker
  NewRad
#  NPScalars_Proca
  ProcaBase
#  Proca_simpleID
  Proca_EMG_ID
  ProcaEvolve
 # PunctureTracker
  QuasiLocalMeasures
  ReflectionSymmetry
 # RotatingSymmetry180
  Slab
  SpaceMask
  SphericalSurface
  StaticConformal
  # SummationByParts
  SymBase
  SystemStatistics
  TerminationTrigger
  Time
  TimerReport
  TmunuBase
  # TwoPunctures
  # TwoPunctures_KerrProca
  # Vectors
  # WeylScal4
"
#------------------------------------------------------------------------------


# Grid setup
#------------------------------------------------------------------------------

CartGrid3D::type                     = "coordbase"
Carpet::domain_from_coordbase        = yes
CoordBase::domainsize                = "minmax"

# make sure all (xmax - xmin)/dx are integers!
CoordBase::xmin                      = -128.00
CoordBase::ymin                      = -128.00
CoordBase::zmin                      =    0.00
CoordBase::xmax                      = +128.00
CoordBase::ymax                      = +128.00
CoordBase::zmax                      = +128.00
CoordBase::dx                        =    1.6
CoordBase::dy                        =    1.6
CoordBase::dz                        =    1.6

driver::ghost_size                   = 3

CoordBase::boundary_size_x_lower     = 3
CoordBase::boundary_size_y_lower     = 3
CoordBase::boundary_size_z_lower     = 3
CoordBase::boundary_size_x_upper     = 3
CoordBase::boundary_size_y_upper     = 3
CoordBase::boundary_size_z_upper     = 3

CoordBase::boundary_shiftout_x_lower = 0
CoordBase::boundary_shiftout_y_lower = 0
CoordBase::boundary_shiftout_z_lower = 1

CarpetRegrid2::symmetry_rotating180  = no

ReflectionSymmetry::reflection_x     = no
ReflectionSymmetry::reflection_y     = no
ReflectionSymmetry::reflection_z     = yes
ReflectionSymmetry::avoid_origin_x   = yes
ReflectionSymmetry::avoid_origin_y   = yes
ReflectionSymmetry::avoid_origin_z   = no


# Mesh refinement
#------------------------------------------------------------------------------

Carpet::max_refinement_levels           = 7

CarpetRegrid2::num_centres              = 1

CarpetRegrid2::num_levels_1             = 7
CarpetRegrid2::radius_1[1]              = 64.0
CarpetRegrid2::radius_1[2]              = 32.0
CarpetRegrid2::radius_1[3]              = 16.0
CarpetRegrid2::radius_1[4]              =  8.0
CarpetRegrid2::radius_1[5]              =  4.0
CarpetRegrid2::radius_1[6]              =  2.0
#CarpetRegrid2::radius_1[7]              =  1.0
#CarpetRegrid2::radius_1[8]              =  0.75


Carpet::use_buffer_zones                = yes
Carpet::prolongation_order_space        = 5
Carpet::prolongation_order_time         = 2

CarpetRegrid2::freeze_unaligned_levels  = yes
CarpetRegrid2::regrid_every             = 0

CarpetRegrid2::verbose                  = yes

Carpet::grid_structure_filename         = "carpet-grid-structure"
Carpet::grid_coordinates_filename       = "carpet-grid-coordinates"

Carpet::time_refinement_factors         = "[1, 1, 1, 1, 1, 1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512]"
Time::dtfac                             = 0.0078125


# Initial Data
#------------------------------------------------------------------------------

ProcaBase::mu                         = 0.0

ADMBase::initial_data                 = "Proca_EMG_RN"
ADMBase::initial_lapse                = "psi^n"
ADMBase::initial_shift                = "zero"
ADMBase::initial_dtlapse              = "zero"
ADMBase::initial_dtshift              = "zero"

ADMBase::lapse_timelevels             = 3
ADMBase::shift_timelevels             = 3
ADMBase::metric_timelevels            = 3

Proca_EMG_ID::initial_lapse_psi_exponent  = -2.0

Proca_EMG_ID::initial_Aphi            = "zero"

Proca_EMG_ID::M                       = 1.0
Proca_EMG_ID::Q                       = 0.2
Proca_EMG_ID::x0                      = 1.0d-5


InitBase::initial_data_setup_method   = "init_some_levels"
Carpet::init_fill_timelevels          = yes
Carpet::init_3_timelevels             = no


# Evolution
#------------------------------------------------------------------------------

TmunuBase::stress_energy_storage  = yes
TmunuBase::stress_energy_at_RHS   = yes
TmunuBase::timelevels             = 1
TmunuBase::prolongation_type      = "none"

ProcaBase::evolution_method       = "ProcaEvolve"

ADMBase::evolution_method               = "LeanBSSNMoL"
ADMBase::lapse_evolution_method         = "LeanBSSNMoL"
ADMBase::shift_evolution_method         = "LeanBSSNMoL"
ADMBase::dtlapse_evolution_method       = "LeanBSSNMoL"
ADMBase::dtshift_evolution_method       = "LeanBSSNMoL"

LeanBSSNMoL::impose_conf_fac_floor_at_initial  = yes
LeanBSSNMoL::eta_beta                     = 1
LeanBSSNMoL::eta_transition               = no
#LeanBSSNMoL::eta_transition_r             = 60
LeanBSSNMoL::conf_fac_floor               = 1.0d-04
LeanBSSNMoL::derivs_order                 = 4
LeanBSSNMoL::use_advection_stencils       = yes
LeanBSSNMoL::calculate_constraints        = yes

ProcaEvolve::calculate_constraints        = yes
ProcaEvolve::Zeta_BC_type                 = "scalar"


# Spatial finite differencing
#------------------------------------------------------------------------------

Dissipation::epsdis = 0.2
Dissipation::order  = 5
Dissipation::vars   = "
  ADMBase::lapse
  ADMBase::shift
  LeanBSSNMoL::conf_fac
  LeanBSSNMoL::hmetric
  LeanBSSNMoL::hcurv
  LeanBSSNMoL::trk
  LeanBSSNMoL::gammat
  ProcaBase::Ei
  ProcaBase::Ai
  ProcaBase::Aphi
  ProcaBase::Zeta
"


# Integration method
#------------------------------------------------------------------------------

MoL::ODE_Method                 = "RK4"
MoL::MoL_Intermediate_Steps     = 4
MoL::MoL_Num_Scratch_Levels     = 1

Carpet::num_integrator_substeps = 4


# Spherical surfaces
#------------------------------------------------------------------------------

SphericalSurface::nsurfaces = 1
SphericalSurface::maxntheta = 66
SphericalSurface::maxnphi   = 124
SphericalSurface::verbose   = no

# Horizon 1
SphericalSurface::ntheta            [0] = 41
SphericalSurface::nphi              [0] = 80
SphericalSurface::nghoststheta      [0] = 2
SphericalSurface::nghostsphi        [0] = 2
CarpetMask::excluded_surface        [0] = 0
CarpetMask::excluded_surface_factor [0] = 1.0

CarpetMask::verbose = no



# # Wave extraction
# #------------------------------------------------------------------------------

# Multipole::nradii       = 3
# Multipole::radius[0]    = 30
# Multipole::radius[1]    = 80
# Multipole::radius[2]    = 100
# Multipole::ntheta       = 120
# Multipole::nphi         = 240
# Multipole::variables    = "
#   NPScalars::psi4re{sw=-2 cmplx='NPScalars::psi4im' name='Psi4'}
# "
# Multipole::integration_method = "Simpson"

# Multipole::out_every    = 128
# Multipole::l_max        = 4
# Multipole::output_hdf5  = no
# Multipole::output_ascii = yes


# Horizons
#------------------------------------------------------------------------------

# AHFinderDirect::verbose_level                           = "algorithm highlights"
AHFinderDirect::verbose_level                            = "physics details"
AHFinderDirect::output_BH_diagnostics                    = "true"
AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES       = no

AHFinderDirect::N_horizons                               = 1
AHFinderDirect::find_every                               = 128

AHFinderDirect::output_h_every                           = 0
AHFinderDirect::max_Newton_iterations__initial           = 50
AHFinderDirect::max_Newton_iterations__subsequent        = 50
AHFinderDirect::max_allowable_Theta_growth_iterations    = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name               = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars               = "order=4"
AHFinderDirect::surface_interpolator_name                = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars                = "order=4"

AHFinderDirect::move_origins                             = no

AHFinderDirect::origin_x                             [1] = 0
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = 0
AHFinderDirect::initial_guess__coord_sphere__radius  [1] = 0.3
AHFinderDirect::which_surface_to_store_info          [1] = 0
AHFinderDirect::set_mask_for_individual_horizon      [1] = no
AHFinderDirect::reset_horizon_after_not_finding      [1] = no
AHFinderDirect::max_allowable_horizon_radius         [1] = 5
#AHFinderDirect::dont_find_after_individual_time      [1] = 30.0


# Isolated Horizons
#-------------------------------------------------------------------------------

QuasiLocalMeasures::verbose                = yes
QuasiLocalMeasures::veryverbose            = no
QuasiLocalMeasures::interpolator           = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options   = "order=4"
QuasiLocalMeasures::spatial_order          = 4
QuasiLocalMeasures::num_surfaces           = 1
QuasiLocalMeasures::surface_index      [0] = 0


# Check for NaNs
#-------------------------------------------------------------------------------

Carpet::poison_new_timelevels = yes
CarpetLib::poison_new_memory  = yes
Carpet::check_for_poison      = no

NaNChecker::check_every     = 512
NanChecker::check_after     = 0
NaNChecker::report_max      = 10
# NaNChecker::verbose         = "all"
NaNChecker::action_if_found = "terminate"
NaNChecker::out_NaNmask     = yes
NaNChecker::check_vars      = "
  LeanBSSNMoL::conf_fac
"


# Timers
#-------------------------------------------------------------------------------

Cactus::cctk_timer_output               = "full"
TimerReport::out_every                  = 5120
TimerReport::n_top_timers               = 40
TimerReport::output_all_timers_together = yes
TimerReport::output_all_timers_readable = yes
TimerReport::output_schedule_timers     = no


# I/O thorns
#-------------------------------------------------------------------------------

Cactus::cctk_run_title       = $parfile
IO::out_dir                  = $parfile

IOScalar::one_file_per_group = yes
IOASCII::one_file_per_group  = yes

IOHDF5::use_checksums        = no
IOHDF5::one_file_per_group   = no

IOBasic::outInfo_every       = 1
IOBasic::outInfo_reductions  = "minimum maximum"
IOBasic::outInfo_vars        = "
  Carpet::physical_time_per_hour
  LeanBSSNMoL::conf_fac
  SystemStatistics::maxrss_mb
"

# for scalar reductions of 3D grid functions
IOScalar::outScalar_every               = 1
IOScalar::outScalar_reductions          = "minimum maximum average norm2"
IOScalar::outScalar_vars                = "
  ADMBase::lapse
  LeanBSSNMoL::ham
  LeanBSSNMoL::mom
  ProcaBase::Zeta
  ProcaEvolve::gc
"

# # output just at one point (0D)
# IOASCII::out0D_every    = 64
# IOASCII::out0D_point_x  = 4.0
# IOASCII::out0D_point_y  = 4.0
# IOASCII::out0D_vars  = "
#   ADMBase::lapse
#   ADMBase::shift
#   Carpet::timing
#   QuasiLocalMeasures::qlm_scalars{out_every = 256}
# "

# IOASCII::output_symmetry_points = no
# IOASCII::out3D_ghosts           = no

# 1D text output
IOASCII::out1D_every            = 1
IOASCII::out1D_d                = no
IOASCII::out1D_x                = yes
IOASCII::out1D_y                = no
IOASCII::out1D_z                = yes
IOASCII::out1D_vars             = "
  ADMBase::lapse
  ADMBase::shift
  ProcaBase::Ei
  ProcaBase::Ai
  ProcaBase::Aphi
  ProcaBase::Zeta
  ProcaEvolve::gc
#  LeanBSSNMoL::hmetric
#  LeanBSSNMoL::trk
  LeanBSSNMoL::conf_fac
  LeanBSSNMoL::ham
  LeanBSSNMoL::mom
"

# 1D HDF5 output
#IOHDF5::out1D_every            = 256
#IOHDF5::out1D_d                = no
#IOHDF5::out1D_x                = yes
#IOHDF5::out1D_y                = no
#IOHDF5::out1D_z                = no
#IOHDF5::out1D_vars             = "
#  ADMBase::lapse
#"

# # 2D HDF5 output
# IOHDF5::out2D_every             = 512
# IOHDF5::out2D_xy                = yes
# IOHDF5::out2D_xz                = no
# IOHDF5::out2D_yz                = yes
# IOHDF5::out2D_vars              = "
#   ADMBase::lapse
#   ProcaBase::Aphi
#   LeanBSSNMoL::ham
# #  LeanBSSNMoL::mom
#   LeanBSSNMoL::conf_fac
# "

# # 3D HDF5 output
# IOHDF5::out_every                      = 8192
# IOHDF5::out_vars                       = "
#   ADMBase::lapse
# "

Carpet::verbose                    = no
Carpet::veryverbose                = no
Carpet::schedule_barriers          = no
Carpet::storage_verbose            = no
CarpetLib::output_bboxes           = no

Cactus::cctk_full_warnings         = yes
Cactus::highlight_warning_messages = no


# Checkpointing and recovery
#-------------------------------------------------------------------------------

CarpetIOHDF5::checkpoint             = yes
IO::checkpoint_dir                   = "../checkpoints"
IO::checkpoint_ID                    = no
IO::checkpoint_every_walltime_hours  = 23
IO::checkpoint_on_terminate          = yes
IO::out_proc_every                   = 2
IO::checkpoint_keep                  = 1

IO::recover                          = "autoprobe"
IO::recover_dir                      = "../checkpoints"

IO::abort_on_io_errors                      = yes
CarpetIOHDF5::open_one_input_file_at_a_time = yes
CarpetIOHDF5::compression_level             = 9


# Run termination
#-------------------------------------------------------------------------------

TerminationTrigger::max_walltime                 = 72 # hours
TerminationTrigger::on_remaining_walltime        = 60 # minutes
TerminationTrigger::output_remtime_every_minutes = 60
TerminationTrigger::termination_from_file        = yes
TerminationTrigger::termination_file             = "terminate.txt"
TerminationTrigger::create_termination_file      = yes

Cactus::terminate       = "time"
Cactus::cctk_final_time = 100.0


More information about the Users mailing list