[Users] Issues Running BNS and GW150914 Example Simulations + storing BSSN variables

Sandeep Suresh CrangaNORE cranganore at ml.jku.at
Mon Sep 15 17:25:31 CDT 2025


Hello Roland,

Many thanks for your detailed email and the potential fixes. I shall 
intersperse through your message:

*BNS:*

<< Comparing your parameter file with the one available on the Einstein 
<< Toolkit page I notice some more changes that were done to it. For << 
example CoordBase::ymin has been changed (to 0.0) and so has the << 
resolution (coarser). << Seeting ymin to 0.0 but keeping the 
RotatingSymmetry180 thorn active is << almost certainly incorrect. << 
Can you confirm that the unmodified bns.par file downloaded directly 
from Apologies --the par file I sent was the one on which I tried some 
changes. For the current runs I am using the unmodified bns.par file, 
exactly as in https://einsteintoolkit.org/gallery/bns/index.html . Now, 
I no longer get the cactus_sim: grille3d.C:125: Grille3d::Grille3d(int, 
int, int, int, int, int, int): Assertion `nr > 0' failed error as 
earlier, but the following new error (cf. attached err file) fails my 
runs: ```Rank 0 with PID 3948578 received signal 11 Writing backtrace to 
bns/backtrace.0.txt 
/system/user/crangano/simulations/bns/output-0000/SIMFACTORY/RunScript: 
line 36: 3948578 Segmentation fault (core dumped) 
/system/user/crangano/simulations/bns/SIMFACTORY/exe/cactus_sim -L 3 
/system/user/crangano/simulations/bns/output-0000/bns.par``` Yes, the 
Lorene codes are the same ones from EinsteinToolkit and not using a copy 
compiled by me. Pls again find the par, err and the out files for the 
same. *GW150914: *<< That is an error that comes out of Simfactory. 
Since the `rpar` file is << a Python script Simfactory will execute it 
as a Python code to get the << par file. I am not sure what you exactly 
mean here. I use python3 to run the rpar file and got the corresponding 
GW150914.par file from that. << Since you are attaching a par file I 
assume that this worked at least once. Is this all on your workstation? 
In that case I am somewhat unsure why there would be a difference 
between running manually and running inside of a job. You only have one 
instance of Python installed, via your OS's package manager (so no conda 
or similar additional package manager), no Python virtualenv that you 
are using? No, for this I am not using an venv or conda environment. The 
only thing that I export additionally is the correct mpi version on my 
clusters (check if the correct mpirun exists) and then just run

<< I notice that that the parfile that you are attaching differs from 
the << one produced by the (current) GW150914.rpar on the gallery 
example << page. Namely it changes the `reflection_z` parameter (and 
nothing << else). Similar to the Rotating180 issue above, you most 
likely have to << adjust some other parameters as well (eg the 
Coordinates::symmetry << option). Yes -- again I sent the tweaked one 
bymistake. The GW150914.rpar is again the same one found on 
EinsteinToolkit Gallery example -- 
https://bitbucket.org/einsteintoolkit/einsteinexamples/raw/master/par/GW150914/GW150914.rpar 
If I don't tweak the rpar file, then I get the following error: ``` 
Traceback (most recent call last): File 
"/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", 
line 126, in <module> sphere_outer_radius = int((outermost_detector + 
final_time)/(i*hr))*i*hr ZeroDivisionError: float division by zero 
Error: Error while executing parameter file script 
/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar 
Aborting Simfactory.``` 
################################################################################################################################## 
/The only one *caveat* is that I am not using the following procedure to 
build and compile this usecase: /


      Configure SimFactory for your machine:

  * If you are on a cluster that SimFactory supports, run

    simfactory/bin/sim setup

    Hit enter for each question if the default is OK. If you intend to
    run on a cluster that requires an allocation, make sure to configure
    this during setup.
  * If you are not using a cluster supported by SimFactory, see
    Compiling the Einstein Toolkit
    <https://docs.einsteintoolkit.org/et-docs/Compiling_the_Einstein_Toolkit>
    for instructions.

Compile Cactus (on 4 processes concurrently; you are encouraged to 
increase this value if you have more cores available and are impatient):

simfactory/bin/sim build -j 4 --thornlist manifest/einsteintoolkit.th

*But rather this method (suggested by one of the EinsteinToolkit team 
member in some previous emails): * cd par # this will create a file  
GW150914.par, can also use python3 if needed python GW150914.rpar # back 
to the main Cactus directory cd .. # The following should be all in one 
line but the email client may # insert a line break 
utils/Scripts/MakeThornList --master thornlists/einsteintoolkit.th 
<http://einsteintoolkit.th> --output thornlists/GW150914.th 
par/GW150914.par # get rid of partially compiled code rm -r configs/sim 
./simfactory/bin/sim build --thornlist thornlists/GW150914.th With this 
I am able to compile and build, but when I run the simulation it fails 
with the following error 
################################################################################################################################# 
Again for this I attach the corresponding par, err and out files for the 
same. *Storing BSSN variables on disk *<< Unfortunately the only way to 
write out values of the Jacobian or << Hessian would be to actually 
create grid functions for them, since we << only compute those "on the 
fly" as they are required to compute the RHS << of the evolution 
equations. They would be quite memory and disk space << intensive 
though. << If derivatives are required, even during postprocessing then 
usually << they would be re-computed. If you'd like to track when the 
grid has << changed then you could try and monitor Carpet's 
GetRegriddingEpoch << aliased function. Or you could try and schedule a 
routine at the << POSTREGRID bin and use CCTK_OutputVarAsByMethod << 
https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc 
<< to output the coordinates when the grid changes. << Note that for a 
Cartesian simulation (not using Llama) one can compute << the 
coordinates of each point using the "origin" and "delta", "ioffset" << 
and "ioffsetdenom" attributes of the HDF5 datasets. Thanks for showing 
how we can dump the coordinates when grid changes, this is helpful. The 
Jacobian and Hessian are optional for us, we can always re-compute the 
Jacobian and Hessian as a post-processing step with our custom FD 
stencil implementation. Although, the main question is storing the 
metric $gamma_ij$ (3-metric), $\alpha$ (lapse function), $\beta^i$ 
(shift vector), $W$ (conformal factor) and the extrinsic curvature trace 
$K$, so the code snippet/additions on the par file pasted below does the 
trick to store on permanent disk storage right? . IOHDF5::out_vars = " 
ML_BSSN::ML_metric ML_BSSN::ML_lapse ML_BSSN::ML_shift ? (I have avoided 
the W, K in the below codes).

> IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse 
> ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000 
> refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r 
> WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}} 
> WeylScal4::curvIi{refinement_levels={3 5}} 
> WeylScal4::curvJr{refinement_levels={3 5}}
<< You can add out_every inside of the curly braces (instead of using 
say CarpetIOHDF5::out_every which would set this globally) the way you 
see it done for Coordinates. So you already are using that, and likely 
are ok with this capability?

Sure, sounds good -- although, I can only gauge ones the simulations run 
successfully. Many thanks again in advance for your help and happy to 
hear back from you with the potential fixes. Kind regards , Sandeep

On 9/15/25 20:16, Roland Haas wrote:
> Hello Sandeep,
>
>> *BNS: *On running the command ./simfactory/bin/sim create-submit bns --parfile bns.par --procs=<num_procs> --num-threads=<num_threads> --walltime=xx:xx:xx
>>
>> I encounter the error: WARNING level 1 from host panther process 0
>>     in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166:
>>     -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
> The "please update the parameter file" are just warnings that you can
> ignore.
>
> There is no way to write Jacobian or Hessian of the variables on the
> grid to disk. You would have to introduce an explicit grid function
> that stores those derivatives and write it to disk. This would however
> by quite expensive memory and disk storage wise.
>
> Usually derivatives (even for posprocessing data after a simulation)
> are computed on the fly. Since all grid patches are using uniform
> resolution (in the Berger-Oliger mesh refinement scheme used by Carpet)
> can be compute easily in postprocessing using just the information
> stored in the HDF5 files.
>
>> cactus_sim:grille3d.C:125:Grille3d::Grille3d(int, int, int, int, int, int, int): Assertion `nr > 0' failed.
>> Rank 0 with PID 3273101 received signal 6
> THis looks like an error from LORENE to me (since it contains French
> language words). Are you using the version of LORENE included with the
> Einstein Toolkit or are you using a copy that you have compiled
> yourself?
>
>> For this I have attached the par file, err file, the out file (the
>> initial data file is the one that is already there online:
>> G2_I12vs12_D4R33T21_45km.resu.xz
>> <https://einsteintoolkit.org/gallery/bns/G2_I12vs12_D4R33T21_45km.resu.xz>)
>> that I am currently using. I tried to find the `nr` in the par file,
>> but didn't find it. I am not sure if this is a MPI/num procs related
>> problem, or is it something in the .par file that one needs to be
>> change ?
> Since this error originates from within LORENE (which has its own grid
> that it used to solve the initial data constraint problem) that
> parameter "nr" is internal to LORENE and not handled by the Einstein
> Toolkit at all.
>
> Comparing your parameter file with the one available on the Einstein
> Toolkit page I notice some more changes that were done to it. For
> exampleCoordBase::ymin has been changed (to 0.0) and so has the
> resolution (coarser).
>
> Seeting ymin to 0.0 but keeping the RotatingSymmetry180 thorn active is
> almost certainly incorrect.
>
> Can you confirm that the unmodified bns.par file downloaded directly
> from
>
> https://einsteintoolkit.org/gallery/bns/index.html
>
> also fails for you? If not then I would try and slowly change one
> parameter at a time to isolate the direct cause of the issue.
>
>> *GW150914: */Error 1 -- /Here, it is much more complicated to understand for me. This persists even after disabling |CoordinatesSymmetry::reflection_z|. Is it likely caused by coarse grid resolution relative to the number of symmetry/ghost zones.
>>
>> /Error 2/ -- Another error that I often encounter is File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module>
>> sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
>> ZeroDivisionError: float division by zero
>> Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar
> That is an error that comes out of Simfactory. Since the `rpar` file is
> a Python script Simfactory will execute it as a Python code to get the
> par file.
>
> Since you are attaching a par file I assume that this worked at least
> once. Is this all on your workstation? In that case I am somewhat
> unsure why there would be a difference between running manually and
> running inside of a job. You only have one instance of Python
> installed, via your OS's package manager (so no conda or similar
> additional package manager), no Python virtualenv that you are using?
>
> I notice that that the parfile that you are attaching differs from the
> one produced by the (current) GW150914.rpar on the gallery example
> page. Namely it changes the `reflection_z` parameter (and nothing
> else). Similar to the Rotating180 issue above, you most likely have to
> adjust some other parameters as well (eg theCoordinates::symmetry
> option).
>
>
>> Aborting Simfactory. -- I guess this occurs in the following line of the *GW150914.rpar*
>>
>> sphere_outer_radius=int((outermost_detector+final_time)/(i*hr))*i*hr
>> sphere_outer_radius=int(sphere_outer_radius/hr) *hr+hr# round up to a multiple of hr
>>
>> So, I changed line 78 of the *GW150914.rpar as follows: *
>>
>> # Number of cells across finest grid radius
>> n=int("@N@") if"@N@"[0] !="@"else28
>> i=max(int(n/4), 1)
>>
>> Is this a valid fix, or does this affect the simulations ?
> If your "n" was less than 4 so that i was 0 then your simulation will
> fail due to being way underresolved. The minimum sensible n is
> somewhere close to 24.
>
> Could you provide your GW150915.rpar file, please?
>
>> For this too, I attach the err, out and the par file such that you can inspect it.
>>
>> *Storing BSSN variables*:We would like to store the BSSN evolved
>> *3-metric* (|γ_ij|), *lapse* (|α|), and *shift* (|β^i|) and also the
>> corresponding coordinates (t, x^i) of these quantities  at regular
>> intervals (e.g., every 128 steps) to manage disk usage efficiently.
>> Moreover, since AMR is used, is there any way we can keep track of
>> the changes in resolution of the coordinates, since we also aim to do
>> spatial derivatives via our FD stencils -- Jacobians and Hessians of
>> these quantities. If there are ways to directly dump the Jacobians
>> and Hessians of these quantities (by adding lines on the par file),
>> w/o us implementing (since we are not aware of the AMR being used at
>> different regions to implement our FD stencils), that would be very
>> useful too.
> Unfortunately the only way to write out values of the Jacobian or
> Hessian would be to actually create grid functions for them, since we
> only compute those "on the fly" as they are required to compute the RHS
> of the evolution equations. They would be quite memory and disk space
> intensive though.
>
> If derivatives are required, even during postprocessing then usually
> they would be re-computed. If you'd like to track when the grid has
> changed then you could try and monitor Carpet's GetRegriddingEpoch
> aliased function. Or you could try and schedule a routine at the
> POSTREGRID bin and use CCTK_OutputVarAsByMethod
>
> https://einsteintoolkit.org/referencemanual/ReferenceManual.html#x1-148000doc
>
> to output the coordinates when the grid changes.
>
> Note that for a Cartesian simulation (not using Llama) one can compute
> the coordinates of each point using the "origin" and "delta", "ioffset"
> and "ioffsetdenom" attributes of the HDF5 datasets.
>
>> IOHDF5::out_vars = " ML_BSSN::ML_metric ML_BSSN::ML_lapse
>> ML_BSSN::ML_shift Grid::Coordinates{out_every=1000000000
>> refinement_levels={0}} ML_BSSN::ML_log_confac WeylScal4::Psi4r
>> WeylScal4::Psi4i WeylScal4::curvIr{refinement_levels={3 5}}
>> WeylScal4::curvIi{refinement_levels={3 5}}
>> WeylScal4::curvJr{refinement_levels={3 5}}
> You can add out_every inside of the curly braces (instead of using say
> CarpetIOHDF5::out_every which would set this globally) the way you see
> it done for Coordinates. So you already are using that, and likely
> are ok with this capability?
>
> Yours,
> Roland
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250916/aa56ea6e/attachment-0001.htm>
-------------- next part --------------
# Carpet parameter file for binary Neutron star system
# physical ID is LORENE dataset G2_I12vs12_D4R33T21_45km.resu
#

#------------------------------------------------------------------------------
# Cactus parameters:
#------------------------------------------------------------------------------
Cactus::cctk_run_title     = "Meudon BNS"
Cactus::cctk_full_warnings = "yes"
Cactus::highlight_warning_messages = "no"

Cactus::terminate       = "time"
Cactus::cctk_final_time = 2500.0

#------------------------------------------------------------------------------
# Activate all necessary thorns:
#------------------------------------------------------------------------------

ActiveThorns = "Boundary CartGrid3D CoordBase Fortran InitBase IOUtil LocalReduce SymBase Time"
ActiveThorns = "AEILocalInterp"
ActiveThorns = "MoL Slab SpaceMask SphericalSurface"
ActiveThorns = "Carpet CarpetInterp CarpetInterp2 CarpetIOASCII CarpetIOHDF5 CarpetIOScalar CarpetLib CarpetIOBasic CarpetReduce CarpetRegrid2 CarpetSlab CarpetTracker CarpetMask LoopControl"
ActiveThorns = "Formaline"
ActiveThorns = "HTTPD Socket"
ActiveThorns = "NaNChecker TerminationTrigger TimerReport"
ActiveThorns = "ADMbase ADMcoupling ADMmacros CoordGauge StaticConformal"
ActiveThorns = "RotatingSymmetry180 ReflectionSymmetry"
ActiveThorns = "Constants TmunuBase HydroBase "
ActiveThorns = "QuasiLocalMeasures"
ActiveThorns = "EOS_Omni"
ActiveThorns = "GRHydro"
ActiveThorns = "SummationByParts"
ActiveThorns = "GenericFD NewRad"
ActiveThorns = "ML_BSSN ML_BSSN_Helper ML_ADMConstraints"
ActiveThorns = "Hydro_Analysis NSTracker"
ActiveThorns = "Dissipation"
ActiveThorns = "SystemStatistics SystemTopology"

# Wave extraction (Psi4)
ActiveThorns = "WeylScal4 Multipole"


#------------------------------------------------------------------------------
# Diagnostic parameters:
#------------------------------------------------------------------------------

Carpet::output_timers_every = 0
Carpet::storage_verbose   = "no"
Carpet::verbose           = "no"
Carpet::veryverbose       = "no"
Carpet::grid_structure_filename   = "carpet-grid-structure"
Carpet::grid_coordinates_filename = "carpet-grid-coordinates"

CarpetLib::output_bboxes  = "no"

CarpetMask::verbose    = "no"
CarpetReduce::verbose  = "no"
CarpetRegrid2::verbose = "no"
CarpetRegrid2::veryverbose    = "no"
CarpetTracker::verbose = "no"


TimerReport::out_every    = 4096
TimerReport::out_filename = "TimerReport"
TimerReport::output_all_timers          = "yes"
TimerReport::output_all_timers_together = "yes"
TimerReport::output_all_timers_readable = "yes"
TimerReport::n_top_timers               = 40


QuasiLocalMeasures::verbose   = "no"
SphericalSurface::verbose   = "no"

#------------------------------------------------------------------------------
# Utility parameters:
#------------------------------------------------------------------------------

NaNChecker::check_every    =  128 # twice for every_coarse
NaNChecker::check_vars = "
            ADMBase::curv 
            ADMBase::metric 
            ADMBase::lapse 
            ADMBase::shift 
            HydroBase::rho 
            HydroBase::eps 
            HydroBase::press 
            HydroBase::vel
"
NaNChecker::action_if_found   =  "terminate"
#NaNChecker::action_if_found = "just warn" #"terminate", "just warn", "abort"

#------------------------------------------------------------------------------
# Run parameters:
#------------------------------------------------------------------------------

#------
# Grid:
#------

MoL::ODE_Method             = "rk4"
MoL::MoL_Intermediate_Steps = 4
MoL::MoL_Num_Scratch_Levels = 1
# use dt = 0.4 dx (works for core collapse)
Time::dtfac = 0.4



ActiveThorns = "CoordBase"

CoordBase::domainsize = "minmax"

CoordBase::xmin =    0.00
CoordBase::ymin = -400.00
CoordBase::zmin =    0.00
CoordBase::xmax = +400.00
CoordBase::ymax = +400.00
CoordBase::zmax = +400.00
CoordBase::dx   =    8.00
CoordBase::dy   =    8.00
CoordBase::dz   =    8.00

CoordBase::boundary_size_x_lower     = 3
CoordBase::boundary_size_y_lower     = 3
CoordBase::boundary_size_z_lower     = 3
CoordBase::boundary_size_x_upper     = 3
CoordBase::boundary_size_y_upper     = 3
CoordBase::boundary_size_z_upper     = 3

CoordBase::boundary_shiftout_x_lower = 1
CoordBase::boundary_shiftout_y_lower = 0
CoordBase::boundary_shiftout_z_lower = 1

ReflectionSymmetry::reflection_z   = "yes"
ReflectionSymmetry::avoid_origin_z = "no"

CartGrid3D::type = "coordbase"
Carpet::domain_from_coordbase = "yes"

Driver::ghost_size                      = 3


# General Carpet parameters:
Carpet::enable_all_storage       = "no"
Carpet::use_buffer_zones         = "yes"
Carpet::schedule_barriers        = "no"

Carpet::poison_new_timelevels    = "yes"
Carpet::check_for_poison         = "no"

Carpet::init_3_timelevels        = "no"
Carpet::init_fill_timelevels     = "yes"

CarpetLib::poison_new_memory         = "yes"
CarpetLib::poison_value              = 114
CarpetLib::check_bboxes              = "no"
CarpetLib::interleave_communications = "yes"
CarpetLib::combine_sends             = "yes"


CarpetInterp::tree_search = "yes"
CarpetInterp::check_tree_search = "no"

CarpetRegrid2::freeze_unaligned_levels = "yes"
CarpetRegrid2::freeze_unaligned_parent_levels = "yes"
CarpetRegrid2::ensure_proper_nesting   = "yes"
CarpetRegrid2::snap_to_coarse          = "yes"
CarpetRegrid2::symmetry_rotating180    = "yes"

# System specific Carpet parameters:
Carpet::max_refinement_levels    = 9 
Carpet::prolongation_order_space = 5
Carpet::prolongation_order_time  = 2
Carpet::refinement_centering     = "vertex"

CarpetRegrid2::regrid_every = 2 # as often as required 
CarpetRegrid2::num_centres  = 3

# box sizes are given by:
# * the stars seem to puff up to about 13M during the initial phase of the evolution
# * I need 12 buffer points (RK4, 3 ghost zones)
# * need three coarse points for interpolation onto last fine buffer point
# these boxes are minimal in this sense. The coarser grid are completely
# covered by the finer grids and their buffers.
# add 4 coarse grid points in between to have some leeway against roundoff
# grid step sizes are for coarsest anticipated simulation dx = 1.5M
CarpetRegrid2::num_levels_1 = 7
CarpetRegrid2::position_x_1 = 15.1875
CarpetRegrid2::radius_1[1]  =240.0
CarpetRegrid2::radius_1[2]  =120.0
CarpetRegrid2::radius_1[3]  = 60.0
CarpetRegrid2::radius_1[4]  = 26.125
CarpetRegrid2::radius_1[5]  = 17.875
CarpetRegrid2::radius_1[6]  = 13     # star seems about 9.5 intially, then puffs up

CarpetRegrid2::num_levels_2 = 7
CarpetRegrid2::position_x_2 = -15.1875
CarpetRegrid2::radius_2[1]  =240.0
CarpetRegrid2::radius_2[2]  =120.0
CarpetRegrid2::radius_2[3]  = 60.0
CarpetRegrid2::radius_2[4]  = 26.125
CarpetRegrid2::radius_2[5]  = 17.875
CarpetRegrid2::radius_2[6]  = 13

CarpetRegrid2::num_levels_3 = 1
CarpetRegrid2::radius_3[1]  =240.0
CarpetRegrid2::radius_3[2]  =120.0
CarpetRegrid2::radius_3[3]  = 60.0
CarpetRegrid2::radius_3[4]  = 30.0
CarpetRegrid2::radius_3[5]  = 15.0 
CarpetRegrid2::radius_3[6]  =  7.5
CarpetRegrid2::radius_3[7]  =  3.75

CarpetMask::excluded_surface       [0] = 2
CarpetMask::excluded_surface_factor[0] = 1.0

CarpetTracker::surface_name[0] = "Righthand NS"
CarpetTracker::surface_name[1] = "Lefthand NS"

#------
# MODEL:
#------

ActiveThorns = "Meudon_Bin_NS"
HydroBase::initial_hydro         = "Meudon_Bin_NS"
ADMBase::initial_data            = "Meudon_Bin_NS"
ADMBase::initial_lapse           = "Meudon_Bin_NS"
ADMBase::initial_shift           = "zero"
ADMBase::initial_dtlapse         = "Meudon_Bin_NS"
ADMBase::initial_dtshift         = "zero"

# change this to be the full path to he initial data file
Meudon_Bin_NS::filename ="/system/user/crangano/einstein_toolkit/BNS/bns.resu"
# M_ADM      3.251
# separation 45km
# K          123.6
# Gamma      2

EOS_Omni::poly_K = 123.613314525753

# For other (non-Polytropic) EOSs Meudon_Bin_NS::eos_table should be labelled
# e.g. ="Hybrid" for hybrid EOS.
#----------
# Numerics:
#----------

InitBase::initial_data_setup_method = "init_some_levels"

TmunuBase::stress_energy_storage = "yes"
TmunuBase::stress_energy_at_RHS  = "yes"
TmunuBase::timelevels            =  1
TmunuBase::prolongation_type     = "none"
TmunuBase::support_old_CalcTmunu_mechanism = "no"

HydroBase::timelevels            = 3

SpaceMask::use_mask      = "yes"

SphericalSurface::nsurfaces = 5
SphericalSurface::maxntheta = 39
SphericalSurface::maxnphi = 76

SphericalSurface::ntheta      [0] = 39
SphericalSurface::nphi        [0] = 76
SphericalSurface::nghoststheta[0] = 2
SphericalSurface::nghostsphi  [0] = 2
SphericalSurface::name        [0] = "Righthand NS"

SphericalSurface::ntheta      [1] = 39
SphericalSurface::nphi        [1] = 76
SphericalSurface::nghoststheta[1] = 2
SphericalSurface::nghostsphi  [1] = 2
SphericalSurface::name        [1] = "Lefthand NS"

SphericalSurface::ntheta      [3] = 39
SphericalSurface::nphi        [3] = 76
SphericalSurface::nghoststheta[3] = 2
SphericalSurface::nghostsphi  [3] = 2
SphericalSurface::set_spherical[3] = yes
SphericalSurface::radius      [3] = 100
SphericalSurface::name        [3] = "waveextract surface at 100"

SphericalSurface::ntheta      [4] = 39
SphericalSurface::nphi        [4] = 76
SphericalSurface::nghoststheta[4] = 2
SphericalSurface::nghostsphi  [4] = 2
SphericalSurface::set_spherical[4] = yes
SphericalSurface::radius      [4] = 250
SphericalSurface::name        [4] = "waveextract surface at 250"

#-----------
# Evolution:
#-----------

HydroBase::evolution_method      = "GRHydro"

ADMMacros::spatial_order = 4
GRHydro::sources_spatial_order = 4

GRHydro::riemann_solver            = "HLLE"   # Marquina is currently not supported by MP
GRHydro::recon_method              = "ppm"
GRHydro::GRHydro_stencil            = 3
GRHydro::bound                     = "flat"
GRHydro::rho_abs_min               = 1.e-11
GRHydro::GRHydro_atmo_tolerance    = 0.01

GRHydro::c2p_reset_pressure        = "yes"

GRHydro::GRHydro_eos_type           = "General"
GRHydro::GRHydro_eos_table          = "Ideal_Fluid"

# these can save some memory since they prevent MoL from allocating unnecessary
# scratch space for saveandrestore variables
GRHydro::GRHydro_MaxNumSandRVars = 0

GRHydro::use_enhanced_ppm            = "yes"
# Parameters are defaults, which in turn are from Colella & Sekora 2008 and
# McCorquodale & Colella 2011
GRHydro::sync_conserved_only     = "yes"
GRHydro::reconstruct_Wv          = "yes"
GRHydro::c2p_resort_to_bisection = "yes"
GRHydro::use_cxx_code            = "yes"


# MacLachlan evolution parameters

ADMBase::metric_type                    = physical
ADMBase::evolution_method               = ML_BSSN
ADMBase::lapse_evolution_method         = ML_BSSN
ADMBase::shift_evolution_method         = ML_BSSN
ADMBase::dtlapse_evolution_method       = ML_BSSN
ADMBase::dtshift_evolution_method       = ML_BSSN


ML_BSSN::timelevels                     = 3
ML_BSSN::my_initial_data                = "ADMBase"
ML_BSSN::my_initial_boundary_condition  = "extrapolate-gammas"
ML_BSSN::my_boundary_condition          = "none"
ML_BSSN::my_rhs_boundary_condition       = "NewRad"
Boundary::radpower                      = 2 # not really needed I think but who knows what NewRad uses

ML_BSSN::harmonicN           = 1      # 1+log
ML_BSSN::harmonicF           = 2.0    # 1+log
ML_BSSN::ShiftGammaCoeff     = 0.75
ML_BSSN::AlphaDriver         = 0.0
ML_BSSN::BetaDriver          = 1.0
ML_BSSN::LapseAdvectionCoeff = 1.0
ML_BSSN::ShiftAdvectionCoeff = 1.0

ML_BSSN::MinimumLapse = 1.0e-8
ML_BSSN::ML_log_confac_bound = "none"
ML_BSSN::ML_metric_bound     = "none"
ML_BSSN::ML_Gamma_bound      = "none"
ML_BSSN::ML_trace_curv_bound = "none"
ML_BSSN::ML_curv_bound       = "none"
ML_BSSN::ML_lapse_bound      = "none"
ML_BSSN::ML_dtlapse_bound    = "none"
ML_BSSN::ML_shift_bound      = "none"
ML_BSSN::ML_dtshift_bound    = "none"

ML_BSSN::UseSpatialBetaDriver = 1
ML_BSSN::SpatialBetaDriverRadius = 50

ML_BSSN::apply_dissipation   = "never"

Dissipation::epsdis = 0.1
Dissipation::order = 5
Dissipation::vars                       = "
        ML_BSSN::ML_log_confac
        ML_BSSN::ML_metric
        ML_BSSN::ML_trace_curv
        ML_BSSN::ML_curv
        ML_BSSN::ML_Gamma
        ML_BSSN::ML_lapse
        ML_BSSN::ML_shift
        ML_BSSN::ML_dtlapse
        ML_BSSN::ML_dtshift
"

#------------------------------------------------------------------------------
# Output:
#------------------------------------------------------------------------------

IO::out_dir = $parfile

IOBasic::outInfo_every = 1
IOBasic::outInfo_reductions = "maximum"
IOBasic::outInfo_vars  = "
 Carpet::physical_time_per_hour
 HydroBase::rho
 ML_ADMConstraints::ML_Ham
 SystemStatistics::maxrss_mb
 GRHydro::dens{reductions = 'sum maximum'}
 HydroBase::w_lorentz
"

IOScalar::outScalar_every      = 256 # every_coarse
IOScalar::all_reductions_in_one_file = "yes"
IOScalar::one_file_per_group   = "yes"
IOScalar::outScalar_reductions = "minimum maximum average norm1 norm2"
IOScalar::outScalar_vars       = "
 ADMBase::lapse
 ADMBase::shift
 ADMBase::metric
 ADMBase::curv
 HydroBase::rho
 HydroBase::vel
 HydroBase::w_lorentz
 GRHydro::dens{reductions = 'minimum maximum average norm1 norm2 sum'}
 SystemStatistics::process_memory_mb
 SphericalSurface::sf_radius
 ML_ADMConstraints::ML_Ham
"

IOASCII::one_file_per_group     = "yes"
IOASCII::compact_format  = "yes"

IOASCII::out0D_every     = 256 # every_coarse
IOASCII::out0D_vars      = "
 Carpet::timing
 QuasiLocalMeasures::qlm_scalars
 SphericalSurface::sf_active
 SphericalSurface::sf_valid
 SphericalSurface::sf_info
 SphericalSurface::sf_radius
 SphericalSurface::sf_origin
 SphericalSurface::sf_coordinate_descriptors
 Hydro_Analysis::Hydro_Analysis_rho_max_loc
 Hydro_Analysis::Hydro_Analysis_rho_max_origin_distance
"

#Set these IOASCII options for initial data only:
IOASCII::out1D_every     = 0
IOASCII::out1D_d         = "no"
IOASCII::out1D_vars      = "
 HydroBase::rho
 HydroBase::vel
 ADMBase::lapse
 ADMBase::shift
 ADMBase::metric
 ADMBase::curv
 ML_ADMConstraints::ML_Ham
"

CarpetIOHDF5::one_file_per_group             = "no"   # this is required by multipatch
CarpetIOHDF5::open_one_input_file_at_a_time  = "yes"
CarpetIOHDF5::out2D_every                    = 1536   # 6*every coarse
CarpetIOHDF5::out2D_xy                       = "yes"
CarpetIOHDF5::out2D_xz                       = "no"
CarpetIOHDF5::out2D_yz                       = "no"
CarpetIOHDF5::out2D_xyplane_z                = 0.0
CarpetIOHDF5::out2D_vars      = "
  CarpetReduce::weight
  Grid::coordinates
  HydroBase::rho
  HydroBase::vel
  HydroBase::eps
  ADMBase::lapse
  ADMBase::shift
  ADMBase::metric
  ML_ADMConstraints::ML_Ham
#  Hydro_Analysis::Hydro_Analysis_Temperature
 "

IOHDF5::out3D_every = 8192 # = 32*every_coarse
IOHDF5::out3D_vars  = "
 CarpetReduce::weight
 HydroBase::rho
 HydroBase::vel
 HydroBase::eps
 ADMBase::lapse
 ADMBase::shift
 ML_ADMConstraints::ML_Ham
 grid::coordinates
"

#------------------------------------------------------------------------------
# Analysis:
#------------------------------------------------------------------------------
Hydro_Analysis::Hydro_Analysis_comp_rho_max = "true"
Hydro_Analysis::Hydro_Analysis_rho_max_loc_only_positive_x = "true"
Hydro_Analysis::Hydro_Analysis_comp_rho_max_origin_distance = "yes"
Hydro_Analysis::Hydro_Analysis_average_multiple_maxima_locations = "yes"
Hydro_Analysis::Hydro_Analysis_interpolator_name = "Lagrange polynomial interpolation (tensor product)"

NSTracker::NSTracker_SF_Name          = "Righthand NS"
NSTracker::NSTracker_SF_Name_Opposite = "Lefthand NS"
NSTracker::NSTracker_max_distance = 3
NSTracker::NSTracker_verbose = "yes"
NSTracker::NSTracker_tracked_location = "Hydro_Analysis::Hydro_Analysis_rho_max_loc"

QuasiLocalMeasures::num_surfaces   = 2
QuasiLocalMeasures::spatial_order  = 4
QuasiLocalMeasures::interpolator = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options = "order=4"
QuasiLocalMeasures::surface_name[0] = "waveextract surface at 100"
QuasiLocalMeasures::surface_name[1] = "waveextract surface at 250"


################################################################################
################################################################################
# Wave extraction
################################################################################
################################################################################


WeylScal4::offset                    = 1e-8 
WeylScal4::fd_order                  = "4th" 
WeylScal4::verbose                   = 0 

Multipole::nradii = 8
Multipole::out_every = 128
Multipole::radius[0] = 45
Multipole::radius[1] = 70
Multipole::radius[2] = 100
Multipole::radius[3] = 125
Multipole::radius[4] = 150
Multipole::radius[5] = 200
Multipole::radius[6] = 250
Multipole::radius[7] = 300
Multipole::variables = "WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='Psi4'}"
Multipole::l_max = 6


#------------------------------------------------------------------------------
# Checkpoint/Recovery:
#------------------------------------------------------------------------------
IOHDF5::checkpoint                  = "yes"
IO::checkpoint_dir                  = $parfile
IO::checkpoint_ID                   = "yes"
IO::checkpoint_on_terminate         = "yes"
# disable extra con2prim in Post_Recover_Variables to ensure bit-identical
# recovery from checkpoints
MoL::run_MoL_PostStep_in_Post_Recover_Variables = "no"

IO::recover     = "autoprobe"
IO::recover_dir = $parfile


#------------------------------------------------------------------------------
# Control
#------------------------------------------------------------------------------
HTTPD::user     = "cactus"
HTTPD::password = "einstein"

ActiveThorns = "TerminationTrigger"
TerminationTrigger::max_walltime = 0.3333333333333333
# Trigger termination 30 minutes before the walltime is reached
TerminationTrigger::on_remaining_walltime   = 30
TerminationTrigger::termination_from_file   = yes
TerminationTrigger::termination_file        = "terminate.txt"
TerminationTrigger::create_termination_file = yes

ActiveThorns = "Trigger"
Trigger::Trigger_Number = 4

Trigger::Trigger_Checked_Variable[0]="Hydro_Analysis::Hydro_Analysis_rho_max_origin_distance"
Trigger::Trigger_Reduction       [0]=""
Trigger::Trigger_Relation        [0]="<"
Trigger::Trigger_Checked_Value   [0]=10
Trigger::Trigger_Reaction        [0]="steerscalar"
Trigger::Trigger_Steered_Scalar      [0] = "CarpetRegrid2::num_levels[2]" # == num_levels_3
Trigger::Trigger_Steered_Scalar_Value[0] = "7"

Trigger::Trigger_Checked_Variable[1]="ADMBase::alp"
Trigger::Trigger_Reduction       [1]="minimum"
Trigger::Trigger_Relation        [1]="<"
Trigger::Trigger_Checked_Value   [1]=0.1
Trigger::Trigger_Reaction        [1]="steerscalar"
Trigger::Trigger_Steered_Scalar      [1] = "CarpetRegrid2::num_levels[2]" # == num_levels_3
Trigger::Trigger_Steered_Scalar_Value[1] = "8"

Trigger::Trigger_Checked_Variable[2]="SphericalSurface::sf_valid[2]"
Trigger::Trigger_Reduction       [2]=""
Trigger::Trigger_Relation        [2]=">"
Trigger::Trigger_Checked_Value   [2]=0.
Trigger::Trigger_Reaction        [2]="steerscalar"
Trigger::Trigger_Steered_Scalar      [2] = "CarpetRegrid2::num_levels[0]" # == num_levels_1
Trigger::Trigger_Steered_Scalar_Value[2] = "1"

Trigger::Trigger_Checked_Variable[3]="SphericalSurface::sf_valid[2]"
Trigger::Trigger_Reduction       [3]=""
Trigger::Trigger_Relation        [3]=">"
Trigger::Trigger_Checked_Value   [3]=0.
Trigger::Trigger_Reaction        [3]="steerscalar"
Trigger::Trigger_Steered_Scalar      [3] = "CarpetRegrid2::num_levels[1]" # == num_levels_2
Trigger::Trigger_Steered_Scalar_Value[3] = "1"
-------------- next part --------------
+ set -e
+ cd /system/user/crangano/simulations/bns/output-0000-active
+ echo Checking:
+ pwd
+ hostname
+ date
+ echo Environment:
+ export CACTUS_NUM_PROCS=1
+ CACTUS_NUM_PROCS=1
+ export CACTUS_NUM_THREADS=1
+ CACTUS_NUM_THREADS=1
+ export GMON_OUT_PREFIX=gmon.out
+ GMON_OUT_PREFIX=gmon.out
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ sort
+ env
+ echo Starting:
++ date +%s
+ export CACTUS_STARTTIME=1757968321
+ CACTUS_STARTTIME=1757968321
+ '[' 1 = 1 ']'
+ '[' 0 -eq 0 ']'
+ /system/user/crangano/simulations/bns/SIMFACTORY/exe/cactus_sim -L 3 /system/user/crangano/simulations/bns/output-0000/bns.par
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:10:
  -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:25:
  -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:45:
  -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:114:
  -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:132:
  -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:145:
  -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:152:
  -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:159:
  -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166:
  -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:193:
  -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:199:
  -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift.
Rank 0 with PID 3948578 received signal 11
Writing backtrace to bns/backtrace.0.txt
/system/user/crangano/simulations/bns/output-0000/SIMFACTORY/RunScript: line 36: 3948578 Segmentation fault      (core dumped) /system/user/crangano/simulations/bns/SIMFACTORY/exe/cactus_sim -L 3 /system/user/crangano/simulations/bns/output-0000/bns.par
-------------- next part --------------
Simulation name: bns
Running simulation bns
Preparing:
Checking:
/system/user/crangano/simulations/bns/output-0000-active
panther
Mon Sep 15 10:32:01 PM CEST 2025
Environment:
Starting:
INFO (Cactus): Increased logging level from 0 to 3
--------------------------------------------------------------------------------

       10                                  
  1   0101       ************************  
  01  1010 10      The Cactus Code V4.18.0    
 1010 1101 011      www.cactuscode.org     
  1001 100101    ************************  
    00010101                               
     100011     (c) Copyright The Authors  
      0100      GNU Licensed. No Warranty  
      0101                                 
--------------------------------------------------------------------------------

Cactus version:    4.18.0
Compile date:      Sep 15 2025 (21:44:45)
Run date:          Sep 15 2025 (22:32:02+0200)
Run host:          panther (pid=3948578)
Working directory: /system/user/crangano/simulations/bns/output-0000
Executable:        /system/user/crangano/simulations/bns/SIMFACTORY/exe/cactus_sim
Parameter file:    /system/user/crangano/simulations/bns/output-0000/bns.par
--------------------------------------------------------------------------------

Activating thorn Cactus...Success -> active implementation Cactus
Activation requested for 
--->Boundary CartGrid3D CoordBase Fortran InitBase IOUtil LocalReduce SymBase Time AEILocalInterp MoL Slab SpaceMask SphericalSurface Carpet CarpetInterp CarpetInterp2 CarpetIOASCII CarpetIOHDF5 CarpetIOScalar CarpetLib CarpetIOBasic CarpetReduce CarpetRegrid2 CarpetSlab CarpetTracker CarpetMask LoopControl Formaline HTTPD Socket NaNChecker TerminationTrigger TimerReport ADMbase ADMcoupling ADMmacros CoordGauge StaticConformal RotatingSymmetry180 ReflectionSymmetry Constants TmunuBase HydroBase QuasiLocalMeasures EOS_Omni GRHydro SummationByParts GenericFD NewRad ML_BSSN ML_BSSN_Helper ML_ADMConstraints Hydro_Analysis NSTracker Dissipation SystemStatistics SystemTopology WeylScal4 Multipole CoordBase Meudon_Bin_NS TerminationTrigger Trigger<---
Warning: thorn CoordBase already scheduled for activation
Warning: thorn TerminationTrigger already scheduled for activation
Thorn Carpet requests automatic activation of MPI
Thorn Carpet requests automatic activation of Timers
Thorn CarpetIOHDF5 requests automatic activation of HDF5
Thorn CarpetLib requests automatic activation of Vectors
Thorn CarpetLib requests automatic activation of CycleClock
Thorn Formaline requests automatic activation of pthreads
Thorn GRHydro requests automatic activation of EOS_Polytrope
Thorn LoopControl requests automatic activation of hwloc
Thorn Meudon_Bin_NS requests automatic activation of LORENE
Thorn QuasiLocalMeasures requests automatic activation of LAPACK
Thorn QuasiLocalMeasures requests automatic activation of TGRtensor
Thorn EOS_Polytrope requests automatic activation of EOS_Base
Thorn HDF5 requests automatic activation of zlib
Thorn LAPACK requests automatic activation of BLAS
Thorn LORENE requests automatic activation of GSL
Activating thorn ADMbase...Success -> active implementation ADMBase
Activating thorn ADMcoupling...Success -> active implementation ADMCoupling
Activating thorn ADMmacros...Success -> active implementation ADMMacros
Activating thorn AEILocalInterp...Success -> active implementation AEILocalInterp
Activating thorn BLAS...Success -> active implementation BLAS
Activating thorn Boundary...Success -> active implementation boundary
Activating thorn Carpet...Success -> active implementation Driver
Activating thorn CarpetInterp...Success -> active implementation interp
Activating thorn CarpetInterp2...Success -> active implementation interp2
Activating thorn CarpetIOASCII...Success -> active implementation IOASCII
Activating thorn CarpetIOBasic...Success -> active implementation IOBasic
Activating thorn CarpetIOHDF5...Success -> active implementation IOHDF5
Activating thorn CarpetIOScalar...Success -> active implementation IOScalar
Activating thorn CarpetLib...Success -> active implementation CarpetLib
Activating thorn CarpetMask...Success -> active implementation CarpetMask
Activating thorn CarpetReduce...Success -> active implementation reduce
Activating thorn CarpetRegrid2...Success -> active implementation CarpetRegrid2
Activating thorn CarpetSlab...Success -> active implementation Hyperslab
Activating thorn CarpetTracker...Success -> active implementation CarpetTracker
Activating thorn CartGrid3D...Success -> active implementation grid
Activating thorn Constants...Success -> active implementation Constants
Activating thorn CoordBase...Success -> active implementation CoordBase
Activating thorn CoordGauge...Success -> active implementation CoordGauge
Activating thorn CycleClock...Success -> active implementation CycleClock
Activating thorn Dissipation...Success -> active implementation Dissipation
Activating thorn EOS_Base...Success -> active implementation EOS_Base
Activating thorn EOS_Omni...Success -> active implementation EOS_Omni
Activating thorn EOS_Polytrope...Success -> active implementation EOS_2d_Polytrope
Activating thorn Formaline...Success -> active implementation Formaline
Activating thorn Fortran...Success -> active implementation Fortran
Activating thorn GenericFD...Success -> active implementation GenericFD
Activating thorn GRHydro...Success -> active implementation GRHydro
Activating thorn GSL...Success -> active implementation GSL
Activating thorn HDF5...Success -> active implementation HDF5
Activating thorn HTTPD...Success -> active implementation HTTPD
Activating thorn hwloc...Success -> active implementation hwloc
Activating thorn Hydro_Analysis...Success -> active implementation Hydro_Analysis
Activating thorn HydroBase...Success -> active implementation HydroBase
Activating thorn InitBase...Success -> active implementation InitBase
Activating thorn IOUtil...Success -> active implementation IO
Activating thorn LAPACK...Success -> active implementation LAPACK
Activating thorn LocalReduce...Success -> active implementation LocalReduce
Activating thorn LoopControl...Success -> active implementation LoopControl
Activating thorn LORENE...Success -> active implementation LORENE
Activating thorn Meudon_Bin_NS...Success -> active implementation Meudon_Bin_NS
Activating thorn ML_ADMConstraints...Success -> active implementation ML_ADMConstraints
Activating thorn ML_BSSN...Success -> active implementation ML_BSSN
Activating thorn ML_BSSN_Helper...Success -> active implementation ML_BSSN_Helper
Activating thorn MoL...Success -> active implementation MethodOfLines
Activating thorn MPI...Success -> active implementation MPI
Activating thorn Multipole...Success -> active implementation multipole
Activating thorn NaNChecker...Success -> active implementation NaNChecker
Activating thorn NewRad...Success -> active implementation NewRad
Activating thorn NSTracker...Success -> active implementation NSTracker
Activating thorn pthreads...Success -> active implementation PTHREADS
Activating thorn QuasiLocalMeasures...Success -> active implementation QuasiLocalMeasures
Activating thorn ReflectionSymmetry...Success -> active implementation ReflectionSymmetry
Activating thorn RotatingSymmetry180...Success -> active implementation RotatingSymmetry180
Activating thorn Slab...Success -> active implementation Slab
Activating thorn Socket...Success -> active implementation Socket
Activating thorn SpaceMask...Success -> active implementation SpaceMask
Activating thorn SphericalSurface...Success -> active implementation SphericalSurface
Activating thorn StaticConformal...Success -> active implementation StaticConformal
Activating thorn SummationByParts...Success -> active implementation SummationByParts
Activating thorn SymBase...Success -> active implementation SymBase
Activating thorn SystemStatistics...Success -> active implementation SystemStatistics
Activating thorn SystemTopology...Success -> active implementation SystemTopology
Activating thorn TerminationTrigger...Success -> active implementation TerminationTrigger
Activating thorn TGRtensor...Success -> active implementation TGRtensor
Activating thorn Time...Success -> active implementation time
Activating thorn TimerReport...Success -> active implementation timerreport
Activating thorn Timers...Success -> active implementation Timers
Activating thorn TmunuBase...Success -> active implementation TmunuBase
Activating thorn Trigger...Success -> active implementation trigger
Activating thorn Vectors...Success -> active implementation Vectors
Activating thorn WeylScal4...Success -> active implementation WeylScal4
Activating thorn zlib...Success -> active implementation zlib
WARNING level 3 from host panther process 0
  in thorn IOUtil, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/IOUtil/CheckpointRecovery.cc:748:
  -> No HDF5 checkpoint files with basefilename 'checkpoint.chkpt' and file extension '.h5' found in recovery directory 'bns'
--------------------------------------------------------------------------------
  if (recover initial data)
    Recover parameters
  endif

  Startup routines
    [CCTK_STARTUP]
      Carpet::MultiModel_Startup: Multi-model Startup routine
      CycleClock::CycleClock_Setup: Set up CycleClock
      LoopControl::LC_setup: Set up LoopControl
      ML_BSSN_Helper::ML_BSSN_SetGroupTags: [meta] Set checkpointing and prolongation group tags
      GROUP hwloc_startup: hwloc startup group
        hwloc::hwloc_version: Output hwloc version
      Timers::Timer_Startup: Prepare hierarchical timers
      SystemTopology::ST_system_topology: Output and/or modify system topology and hardware locality
      CarpetInterp::CarpetInterpStartup: Startup routine
      CarpetReduce::CarpetReduceStartup: Startup routine
      CartGrid3D::SymmetryStartup: Register GH Extension for GridSymmetry
      CoordBase::CoordBase_Startup: Register a GH extension to store the coordinate system handles
      Carpet::Driver_Startup: Startup routine
      EOS_Omni::EOS_Omni_Startup: [global] Set up conversion factors and other fun stuff
      EOS_Polytrope::EOS_Polytrope_Startup: Setup the polytropic EOS
      Formaline::Formaline_PrintIDs: [meta] Print the build and simulation ids
      GRHydro::GRHydro_Startup: Startup banner
      GRHydro::GRHydro_RegisterMask: Register the hydro masks
      GROUP HTTP_Startup: HTTP daemon startup group
        HTTPD::HTTP_StartServer: [global] Start HTTP server
        GROUP HTTP_SetupPages: Group to setup stuff which needs to be done between starting the server and the first time it serves pages
        HTTPD::HTTP_FirstServ: [global] Serve first pages at startup
      HydroBase::HydroBase_StartUp: Startup banner
      IOUtil::IOUtil_Startup: Startup routine
      LocalReduce::LocalReduce_Startup: Startup routine
      CarpetIOBasic::CarpetIOBasicStartup: [global] Startup routine
      ML_ADMConstraints::ML_ADMConstraints_Startup: [meta] create banner
      ML_BSSN::ML_BSSN_Startup: [meta] create banner
      ML_BSSN_Helper::ML_BSSN_RegisterSlicing: [meta] Register slicing
      CarpetIOHDF5::CarpetIOHDF5_Startup: Startup routine
      MoL::MoL_Startup: Startup banner
      Slab::Slab_InitMPIDatatypes: Create MPI datatypes for complex variables in C
      SymBase::SymBase_Startup: Register GH Extension for SymBase
      CarpetIOScalar::CarpetIOScalarStartup: [global] Startup routine
      TerminationTrigger::TerminationTrigger_StartSignalHandler: Start signal handler
      CarpetIOASCII::CarpetIOASCIIStartup: [global] Startup routine
      Trigger::Trigger_Startup: Startup Routine
      Vectors::Vectors_Startup: Print startup message
      WeylScal4::WeylScal4_Startup: [meta] create banner
      AEILocalInterp::AEILocalInterp_U_Startup: register CCTK_InterpLocalUniform() interpolation operators

  Startup routines which need an existing grid hierarchy
    [CCTK_WRAGH]
      ADMBase::Einstein_InitSymBound: [global] Set up GF symmetries
      Boundary::Boundary_RegisterBCs: [global] Register boundary conditions that this thorn provides
      CarpetRegrid2::CarpetRegrid2_Initialise: [global] Initialise locations of refined regions
      CartGrid3D::RegisterCartGrid3DCoords: [meta] Register coordinates for the Cartesian grid
      CoordGauge::Einstein_ActivateSlicing: Initialize slicing, setup priorities for mixed slicings
      CoordGauge::Einstein_SetNextSlicing: Identify the slicing for the next iteration
      Formaline::Formaline_OutputSource: [meta] Output Cactus source tree
      Formaline::Formaline_RegisterWarnings: [meta] Register to receive warnings and info messages from the flesh
      Formaline::Formaline_AnnounceInitial: [global] Put some meta information about the current run into permanent storage
      GRHydro::GRHydro_ClearLastMoLPostStep: [global] Initialize InLastMoLPostStep to zero
      ML_BSSN_Helper::ML_BSSN_ParamCompat: [meta] Handle parameter backward compatibility
      MoL::MoL_SetupIndexArrays: Set up the MoL bookkeeping index arrays
      MoL::MoL_SetScheduleStatus: [global] Set the flag so it is ok to register with MoL
      TmunuBase::TmunuBase_SetStressEnergyState: [global] Set the stress_energy_state variable
      GROUP MoL_Register: The group where physics thorns register variables with MoL
        GRHydro::GRHydro_Register: Register variables for MoL
        ML_ADMConstraints::ML_ADMConstraints_RegisterVars: [meta] Register Variables for MoL
        ML_BSSN::ML_BSSN_RegisterVars: [meta] Register Variables for MoL
        ML_BSSN_Helper::ML_BSSN_RegisterConstrained: [meta] Register ADMBase variables as constrained
        WeylScal4::WeylScal4_RegisterVars: [meta] Register Variables for MoL
      Slab::Slab_InitTimers: Initialise timers
      SpaceMask::MaskSym: [global] Set grid symmetries for mask
      SpaceMask::MaskSym_emask: [global] Set grid symmetries for emask (compatibility mode)
      GROUP SymBase_Wrapper: Wrapper group for SymBase
        GROUP SymmetryRegister: Register your symmetries here
          CartGrid3D::CartGrid3D_RegisterSymmetryBoundaries: [meta] Register symmetry boundaries
          ML_ADMConstraints::ML_ADMConstraints_RegisterSymmetries: [meta] register symmetries
          ML_BSSN::ML_BSSN_RegisterSymmetries: [meta] register symmetries
          ReflectionSymmetry::ReflectionSymmetry_Register: Register reflection symmetry boundaries
          RotatingSymmetry180::Rot180_RegisterSymmetry: Register symmetry boundaries
          WeylScal4::WeylScal4_RegisterSymmetries: [meta] register symmetries
        SymBase::SymBase_Statistics: Print symmetry boundary face descriptions
      MoL::MoL_ReportNumberVariables: [meta] Report how many of each type of variable there are
  Parameter checking routines
    [CCTK_PARAMCHECK]
      ADMBase::ADMBase_ParamCheck: [global] Check consistency of parameters
      Boundary::Boundary_Check: Check dimension of grid variables
      Carpet::CarpetParamCheck: Parameter checking routine
      CarpetLib::CarpetLib_test_prolongate_3d_rf2: [global] Test prolongation operators
      CarpetMask::CarpetSurfaceParamCheck: [global] Check parameters
      CarpetRegrid2::CarpetRegrid2_ParamCheck: Check parameters
      CartGrid3D::ParamCheck_CartGrid3D: Check coordinates for CartGrid3D
      Dissipation::dissipation_paramcheck: Check dissipation parameters for consistency
      Fortran::CheckFortranParameters: Test whether Fortran parameters work correctly
      GRHydro::GRHydro_ParamCheck: Check parameters
      ML_BSSN_Helper::ML_BSSN_ParamCheck: [meta] Check parameters
      MoL::MoL_ParamCheck: Basic parameter checking
      Multipole::Multipole_ParamCheck: [global] Check Multipole parameters
      QuasiLocalMeasures::qlm_paramcheck: [global] Check quasi-local parameter settings
      RotatingSymmetry180::Rot180_CheckTensorTypes: Check tensor type definitions for consistency
      SphericalSurface::SphericalSurface_ParamCheck: [global] Check that all surface names are unique
      TerminationTrigger::TerminationTrigger_ParamCheck: Check consitency of parameters
      TmunuBase::TmunuBase_ParamCheck: [global] Check that no deprecated parameters are used.
      Trigger::Trigger_ParamCheck: Parameter checking
      Vectors::Vectors_Test: Run correctness tests.

  Initialisation
    if (NOT (recover initial data AND recovery_mode is 'strict'))
      [CCTK_PREREGRIDINITIAL]
      Set up grid hierarchy
      [CCTK_POSTREGRIDINITIAL]
        CartGrid3D::SpatialCoordinates: Set Coordinates after regridding
        GROUP MaskBase_SetupMask: Set up the weight function
          GROUP MaskBase_SetupMaskAll: Set up the weight function
            CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
            CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
            GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
              CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
              CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
            GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
              CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
              CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
            CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
            GROUP SetupMask: Set up the real weight function (schedule other routines in here)
            CarpetReduce::MaskBase_TestMask: [global] Test the weight function
        Dissipation::setup_epsdis: Setup spatially varying dissipation
        SpaceMask::MaskZero: Initialise mask to zero
        GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
        GROUP GRHydro_PrimitiveBoundaries: Apply boundary conditions to all primitives
          GRHydro::GRHydro_SelectPrimitiveBoundaries: [level] Select primitive variables for boundary conditions
          GROUP GRHydro_ApplyPrimitiveBCs: Apply boundary conditions to all primitive variables
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP HydroBase_ExcisionMaskSetup: Set up hydro excision mask
          HydroBase::HydroBase_InitExcisionMask: Initialize hydro excision mask to 'no excision everywhere'
        GROUP MoL_PseudoEvolutionBoundaries: Apply boundary conditions to pseudo-evolved quantities
          GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
            ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
            GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
            ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
            GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
            ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
            GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
            WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
            GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        SpaceMask::MaskOne: Set mask to one
        GRHydro::GRHydro_SetupMask: Initialize the atmosphere mask
      [CCTK_BASEGRID]
        ADMBase::ADMBase_SetShiftStateOn: Set the shift_state variable to 1
        ADMBase::ADMBase_SetDtLapseStateOn: Set the dtlapse_state variable to 1
        ADMBase::ADMBase_SetDtShiftStateOn: Set the dtshift_state variable to 1
        ADMMacros::ADMMacros_SetLocalSpatialOrder: Initialize the local_spatial_order
        CartGrid3D::SpatialSpacings: Set up ranges for spatial 3D Cartesian coordinates (on all grids)
        CartGrid3D::SpatialCoordinates: Set up spatial 3D Cartesian coordinates on the GH
        SphericalSurface::SphericalSurface_SetupRes: [global] [loop-local] Set surface resolution automatically
        Dissipation::dissipation_basegrid: Ensure that there are enough ghost zones
        GRHydro::GRHydro_Reset_Execution_Flags: [global] Initially set execution flags to 'YEAH, Execute'!
        GRHydro::GRHydro_InitSymBound: Schedule symmetries and check shift state
        GRHydro::reset_GRHydro_C2P_failed: Initialise the mask function that contains the points where C2P has failed (at BASEGRID)
        Hydro_Analysis::Hydro_Analysis_Init: [global] Initialize variables
        ML_ADMConstraints::ML_ADMConstraints_CheckBoundaries: [meta] check boundaries treatment
        ML_BSSN::ML_BSSN_CheckBoundaries: [meta] check boundaries treatment
        NaNChecker::NaNChecker_ResetCounter: [global] Reset the NaNChecker::NaNsFound counter
        SpaceMask::MaskZero: Initialise mask to zero
        SpaceMask::MaskOne: Set old style mask to one
        SphericalSurface::SphericalSurface_Setup: [global] Calculate surface coordinate descriptors
        GROUP MaskBase_SetupMask: Set up the weight function
          GROUP MaskBase_SetupMaskAll: Set up the weight function
            CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
            CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
            GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
              CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
              CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
            GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
              CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
              CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
            CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
            GROUP SetupMask: Set up the real weight function (schedule other routines in here)
            CarpetReduce::MaskBase_TestMask: [global] Test the weight function
        SphericalSurface::SphericalSurface_Set: [global] Set surface radii to be used for initial setup in other thorns
        GROUP SphericalSurface_HasBeenSet: Set the spherical surfaces before this group, and use it afterwards
          SphericalSurface::SphericalSurface_CheckState: [global] Test the state of the spherical surfaces
        SummationByParts::SBP_SetNormMask: Setup the mask for the calculation of the norm
        SymBase::SymBase_Check: Check whether the driver set up the grid consistently
        TerminationTrigger::TerminationTrigger_ResetTrigger: Clear trigger state
        TerminationTrigger::TerminationTrigger_StartTimer: Start timer
        TerminationTrigger::TerminationTrigger_CreateFile: Create termination file
        Time::Time_Initialise: [global] Initialise Time variables
        Time::TemporalSpacings: [singlemap] Set timestep based on Courant condition (courant_static)
        WeylScal4::WeylScal4_CheckBoundaries: [meta] check boundaries treatment
      [CCTK_INITIAL]
        StaticConformal::StaticConformal_InitialiseState: Set the conformal_state variable to 0
        GROUP ADMBase_InitialData: Schedule group for calculating ADM initial data
        GRHydro::GRHydro_EOSHandle: [global] Set the EOS number
        CarpetIOASCII::CarpetIOASCIIInit: [global] Initialisation routine
        CarpetIOBasic::CarpetIOBasicInit: [global] Initialisation routine
        CarpetIOHDF5::CarpetIOHDF5_Init: [global] Initialisation routine
        CarpetIOScalar::CarpetIOScalarInit: [global] Initialisation routine
        GRHydro::GRHydro_Rho_Minima_Setup: Set up minimum for the rest-mass density in the atmosphere (before intial data)
        GRHydro::GRHydro_SetupMask: Initialize the atmosphere mask
        GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
        GROUP ADMBase_InitialGauge: Schedule group for the ADM initial gauge condition
          ADMBase::ADMBase_ShiftZero: Set the shift to 0 at all points
          ADMBase::ADMBase_DtShiftZero: Set the dtshift to 0 at all points
        GROUP HydroBase_Initial: HydroBase initial data group
          GROUP GRHydro_Initial: GRHydro initial data group
          GROUP HydroBase_ExcisionMaskSetup: Set up hydro excision mask
            HydroBase::HydroBase_InitExcisionMask: Initialize hydro excision mask to 'no excision everywhere'
          Meudon_Bin_NS::Meudon_Bin_NS_initialise: Set up binary neutron star initial data
        GROUP ADMBase_PostInitial: Schedule group for modifying the ADM initial data, such as e.g. adding noise
        GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
        GRHydro::GRHydro_InitialAtmosphereReset: Use mask to enforce atmosphere at initial time
        ML_BSSN::ML_BSSN_InitialADMBase1Everywhere: ML_BSSN_InitialADMBase1Everywhere
        ML_BSSN::ML_BSSN_InitialADMBase2Interior: ML_BSSN_InitialADMBase2Interior
        ML_BSSN::ML_BSSN_InitialADMBase2BoundaryScalar: ML_BSSN_InitialADMBase2BoundaryScalar
        ML_BSSN_Helper::ML_BSSN_ExtrapolateGammas: Extrapolate Gammas and time derivatives of lapse and shift
        MoL::MoL_StartLoop: [level] Initialise the step size control
        QuasiLocalMeasures::qlm_init: [global] Initialise quasi-local calculations
        GROUP HydroBase_Prim2ConInitial: Recover the conservative variables from the primitive variables
          GRHydro::Primitive2ConservativeCells: Convert initial data given in primive variables to conserved variables
      [CCTK_POSTINITIAL]
        CarpetIOHDF5::CarpetIOHDF5_CloseFiles: [global] Close all filereader input files
        GRHydro::GRHydro_Scalar_Setup: Set up and check scalars for efficiency
        GROUP MoL_PostStepModify: The group for physics thorns to schedule enforcing constraints
          ML_BSSN::ML_BSSN_EnforceEverywhere: ML_BSSN_EnforceEverywhere
        GROUP MoL_PostStep: Ensure that everything is correct after the initial data have been set up
          ML_BSSN::ML_BSSN_SelectBoundConds: [level] select boundary conditions
          GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
          GRHydro::GRHydro_SetLastMoLPostStep: [level] Set grid scalar InLastMoLPostStep if this is the last MoL PostStep call
          GROUP ML_BSSN_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          ML_ADMConstraints::ML_ADMConstraints_SelectBoundConds: [level] select boundary conditions
          GROUP ML_ADMConstraints_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          ML_BSSN::ML_BSSN_ADMBaseInterior: ML_BSSN_ADMBaseInterior
          ML_BSSN::ML_BSSN_ADMBaseBoundaryScalar: ML_BSSN_ADMBaseBoundaryScalar
          ML_BSSN::ML_BSSN_ADMBaseEverywhere: ML_BSSN_ADMBaseEverywhere
          ML_BSSN_Helper::ML_BSSN_ADMBase_SelectBCs: [level] Select boundary conditions for ADMBase variables
          GROUP ML_BSSN_ADMBase_ApplyBCs: Apply boundary conditions to ADMBase variables
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
          GROUP HydroBase_PostStep: Post step tasks for hydro thorns
            GROUP GRHydro_PostStep: Post step tasks for GRHydro
            GROUP GRHydro_AtmosphereMaskBoundaries: Apply boundary conditions to primitives
              GRHydro::GRHydro_SelectAtmosphereMaskBoundaries: [level] Select atmosphere mask for boundary conditions
              GROUP GRHydro_ApplyAtmosphereMaskBCs: Apply boundary conditions to real-valued atmosphere mask
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
            GRHydro::GRHydroPostSyncAtmosphereMask: Set integer atmosphere mask from synchronized real atmosphere mask
            if (GRHydro::InLastMoLPostStep)
              GRHydro::GRHydro_AtmosphereReset: Reset the atmosphere
            if (GRHydro::InLastMoLPostStep)
              GROUP GRHydro_PrimitiveInitialGuessesBoundaries: Apply boundary conditions to those primitives used as initial guesses
                GRHydro::GRHydro_SelectPrimitiveInitialGuessesBoundaries: [level] Select initial guess primitive variables for boudary conditions
                GROUP GRHydro_ApplyPrimitiveInitialGuessBCs: Apply boundary conditions to initial guess primitive variables
                  GROUP BoundaryConditions: Execute all boundary conditions
                    Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                    CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                    ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                    RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                    RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                  Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
            GROUP HydroBase_Boundaries: HydroBase-internal Boundary conditions group
              GROUP Do_GRHydro_Boundaries: GRHydro Boundary conditions group
              GROUP HydroBase_Select_Boundaries: Group to schedule the boundary condition functions
                if (GRHydro::execute_MoL_PostStep)
                  GRHydro::GRHydro_Bound: [level] Select GRHydro boundary conditions
              GROUP HydroBase_ApplyBCs: Apply the boundary conditions of HydroBase
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
            GROUP HydroBase_Con2Prim: Convert from conservative to primitive variables
              if (GRHydro::execute_MoL_Step)
                GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
              if (GRHydro::execute_MoL_PostStep)
                GRHydro::Con2Prim: Convert back to primitive variables (general)
          GROUP SetTmunu: Group for calculating the stress-energy tensor
            TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
            GROUP AddToTmunu: Add to the stress-energy tensor here
              GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
          WeylScal4::WeylScal4_SelectBoundConds: [level] select boundary conditions
          GROUP WeylScal4_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP MoL_PseudoEvolution: Calculate pseudo-evolved quantities
          GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
          GROUP ML_ADMConstraints_evaluate_group: ML_ADMConstraints_evaluate
            ML_ADMConstraints::ML_ADMConstraints_evaluate: ML_ADMConstraints_evaluate
            GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
              ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
              GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ML_BSSN_ConstraintsEverywhere_group: ML_BSSN_ConstraintsEverywhere
            ML_BSSN::ML_BSSN_ConstraintsEverywhere: ML_BSSN_ConstraintsEverywhere
            GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
              ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
              GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ML_BSSN_ConstraintsInterior_group: ML_BSSN_ConstraintsInterior
            ML_BSSN::ML_BSSN_ConstraintsInterior: ML_BSSN_ConstraintsInterior
            GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
              ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
              GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP WeylScal4_psi4_calc_4th_group: WeylScal4_psi4_calc_4th
            WeylScal4::WeylScal4_psi4_calc_4th: WeylScal4_psi4_calc_4th
            GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
              WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
              GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        SummationByParts::SBP_CheckGridSizes: Check grid sizes and ghost zones
      Initialise finer grids recursively
      Restrict from finer grids
      [CCTK_POSTRESTRICTINITIAL]
        GROUP MoL_PostStep: Ensure that everything is correct after restriction
          ML_BSSN::ML_BSSN_SelectBoundConds: [level] select boundary conditions
          GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
          GRHydro::GRHydro_SetLastMoLPostStep: [level] Set grid scalar InLastMoLPostStep if this is the last MoL PostStep call
          GROUP ML_BSSN_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          ML_ADMConstraints::ML_ADMConstraints_SelectBoundConds: [level] select boundary conditions
          GROUP ML_ADMConstraints_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          ML_BSSN::ML_BSSN_ADMBaseInterior: ML_BSSN_ADMBaseInterior
          ML_BSSN::ML_BSSN_ADMBaseBoundaryScalar: ML_BSSN_ADMBaseBoundaryScalar
          ML_BSSN::ML_BSSN_ADMBaseEverywhere: ML_BSSN_ADMBaseEverywhere
          ML_BSSN_Helper::ML_BSSN_ADMBase_SelectBCs: [level] Select boundary conditions for ADMBase variables
          GROUP ML_BSSN_ADMBase_ApplyBCs: Apply boundary conditions to ADMBase variables
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
          GROUP HydroBase_PostStep: Post step tasks for hydro thorns
            GROUP GRHydro_PostStep: Post step tasks for GRHydro
            GROUP GRHydro_AtmosphereMaskBoundaries: Apply boundary conditions to primitives
              GRHydro::GRHydro_SelectAtmosphereMaskBoundaries: [level] Select atmosphere mask for boundary conditions
              GROUP GRHydro_ApplyAtmosphereMaskBCs: Apply boundary conditions to real-valued atmosphere mask
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
            GRHydro::GRHydroPostSyncAtmosphereMask: Set integer atmosphere mask from synchronized real atmosphere mask
            if (GRHydro::InLastMoLPostStep)
              GRHydro::GRHydro_AtmosphereReset: Reset the atmosphere
            if (GRHydro::InLastMoLPostStep)
              GROUP GRHydro_PrimitiveInitialGuessesBoundaries: Apply boundary conditions to those primitives used as initial guesses
                GRHydro::GRHydro_SelectPrimitiveInitialGuessesBoundaries: [level] Select initial guess primitive variables for boudary conditions
                GROUP GRHydro_ApplyPrimitiveInitialGuessBCs: Apply boundary conditions to initial guess primitive variables
                  GROUP BoundaryConditions: Execute all boundary conditions
                    Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                    CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                    ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                    RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                    RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                  Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
            GROUP HydroBase_Boundaries: HydroBase-internal Boundary conditions group
              GROUP Do_GRHydro_Boundaries: GRHydro Boundary conditions group
              GROUP HydroBase_Select_Boundaries: Group to schedule the boundary condition functions
                if (GRHydro::execute_MoL_PostStep)
                  GRHydro::GRHydro_Bound: [level] Select GRHydro boundary conditions
              GROUP HydroBase_ApplyBCs: Apply the boundary conditions of HydroBase
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
            GROUP HydroBase_Con2Prim: Convert from conservative to primitive variables
              if (GRHydro::execute_MoL_Step)
                GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
              if (GRHydro::execute_MoL_PostStep)
                GRHydro::Con2Prim: Convert back to primitive variables (general)
          GROUP SetTmunu: Group for calculating the stress-energy tensor
            TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
            GROUP AddToTmunu: Add to the stress-energy tensor here
              GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
          WeylScal4::WeylScal4_SelectBoundConds: [level] select boundary conditions
          GROUP WeylScal4_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP MoL_PseudoEvolutionBoundaries: Apply boundary conditions to pseudo-evolved quantities
          GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
            ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
            GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
            ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
            GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
            ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
            GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
            WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
            GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
      [CCTK_POSTPOSTINITIAL]
        GRHydro::GRHydro_Rho_Minima_Setup_Final: Set the value of the rest-mass density of the atmosphere which will be used during the evolution
        GRHydro::GRHydro_InitialAtmosphereReset: Use mask to enforce atmosphere at initial time
        GROUP Con2Prim: Convert from conservative to primitive variables (might be redundant)
          if (GRHydro::execute_MoL_Step)
            GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
          if (GRHydro::execute_MoL_PostStep)
            GRHydro::Con2Prim: Convert back to primitive variables (general)
        GROUP SetTmunu: Calculate the stress-energy tensor
          TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
          GROUP AddToTmunu: Add to the stress-energy tensor here
            GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
        GROUP ADMConstraintsGroup: Evaluate ADM constraints, and perform symmetry boundary conditions
      [CCTK_POSTSTEP]
        SphericalSurface::SphericalSurface_Set: [global] Set surface radii
        GROUP SphericalSurface_HasBeenSet: Set the spherical surfaces before this group, and use it afterwards
          SphericalSurface::SphericalSurface_CheckState: [global] Test the state of the spherical surfaces
        GRHydro::GRHydro_RefinementLevel: Calculate current refinement level (for the check of the C2P mask)
        GRHydro::check_GRHydro_C2P_failed: Check the mask function that contains the points where C2P has failed and report an error in case a failure is found
        HTTPD::HTTP_Work: [global] Working routine
        HTTPD::HTTP_ContentWork: [global] Content Working routine
        GROUP HydroBase_ExcisionHasBeenSet: Group to schedule thorns changing the mask before and thorns using the mask after
        GROUP Hydro_Analysis: Group for Hydro_Analysis routines
          Hydro_Analysis::Hydro_Analysis_PrepareReduction: [global] [loop-local] Compute the local reduction results
          Hydro_Analysis::Hydro_Analysis_Reduction: [global] Compute the global reduction results
          GROUP Hydro_Analysis_LocationSearch: Look for the location of the maximum density
            Hydro_Analysis::Hydro_Analysis_LocationSearch_Setup: [global] Prepare data structures for search
            Hydro_Analysis::Hydro_Analysis_LocationSearch_Search: [global] [loop-local] Search for the location of the maximum density
            Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine: [global] Communicate and verify the location of the maximum density
          Hydro_Analysis::Hydro_Analysis_FindSeparation: [global] Compute the proper distance between the maximum of the density and the origin (along a straight coordinate line)
          NSTracker::NSTracker: [global] Look for the star and set grid accordingly
        GROUP zzz_NaNChecker_NaNCheck: Check for NaNs and count them in NaNChecker::NaNsFound
          NaNChecker::NaNChecker_NaNCheck_Prepare: [level] Prepare data structures to check for NaNs
          NaNChecker::NaNChecker_NaNCheck_Check: [local] Check for NaNs
          NaNChecker::NaNChecker_NaNCheck_Finish: [level] Count NaNs in NaNChecker::NaNsFound
        NaNChecker::NaNChecker_TakeAction: [global] [loop-level] Output NaNChecker::NaNmask and take action according to NaNChecker::action_if_found
        SpaceMask::CheckMask: Ensure that all mask values are legal
        Dissipation::setup_epsdis: Setup spatially varying dissipation
        GROUP MaskBase_SetupMask: Set up the weight function
          GROUP MaskBase_SetupMaskAll: Set up the weight function
            CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
            CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
            GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
              CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
              CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
            GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
              CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
              CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
            CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
            GROUP SetupMask: Set up the real weight function (schedule other routines in here)
            CarpetReduce::MaskBase_TestMask: [global] Test the weight function
    endif
    if (recover initial data)
      [CCTK_BASEGRID]
        ADMBase::ADMBase_SetShiftStateOn: Set the shift_state variable to 1
        ADMBase::ADMBase_SetDtLapseStateOn: Set the dtlapse_state variable to 1
        ADMBase::ADMBase_SetDtShiftStateOn: Set the dtshift_state variable to 1
        ADMMacros::ADMMacros_SetLocalSpatialOrder: Initialize the local_spatial_order
        CartGrid3D::SpatialSpacings: Set up ranges for spatial 3D Cartesian coordinates (on all grids)
        CartGrid3D::SpatialCoordinates: Set up spatial 3D Cartesian coordinates on the GH
        SphericalSurface::SphericalSurface_SetupRes: [global] [loop-local] Set surface resolution automatically
        Dissipation::dissipation_basegrid: Ensure that there are enough ghost zones
        GRHydro::GRHydro_Reset_Execution_Flags: [global] Initially set execution flags to 'YEAH, Execute'!
        GRHydro::GRHydro_InitSymBound: Schedule symmetries and check shift state
        GRHydro::reset_GRHydro_C2P_failed: Initialise the mask function that contains the points where C2P has failed (at BASEGRID)
        Hydro_Analysis::Hydro_Analysis_Init: [global] Initialize variables
        ML_ADMConstraints::ML_ADMConstraints_CheckBoundaries: [meta] check boundaries treatment
        ML_BSSN::ML_BSSN_CheckBoundaries: [meta] check boundaries treatment
        NaNChecker::NaNChecker_ResetCounter: [global] Reset the NaNChecker::NaNsFound counter
        SpaceMask::MaskZero: Initialise mask to zero
        SpaceMask::MaskOne: Set old style mask to one
        SphericalSurface::SphericalSurface_Setup: [global] Calculate surface coordinate descriptors
        GROUP MaskBase_SetupMask: Set up the weight function
          GROUP MaskBase_SetupMaskAll: Set up the weight function
            CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
            CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
            GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
              CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
              CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
            GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
              CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
              CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
            CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
            GROUP SetupMask: Set up the real weight function (schedule other routines in here)
            CarpetReduce::MaskBase_TestMask: [global] Test the weight function
        SphericalSurface::SphericalSurface_Set: [global] Set surface radii to be used for initial setup in other thorns
        GROUP SphericalSurface_HasBeenSet: Set the spherical surfaces before this group, and use it afterwards
          SphericalSurface::SphericalSurface_CheckState: [global] Test the state of the spherical surfaces
        SummationByParts::SBP_SetNormMask: Setup the mask for the calculation of the norm
        SymBase::SymBase_Check: Check whether the driver set up the grid consistently
        TerminationTrigger::TerminationTrigger_ResetTrigger: Clear trigger state
        TerminationTrigger::TerminationTrigger_StartTimer: Start timer
        TerminationTrigger::TerminationTrigger_CreateFile: Create termination file
        Time::Time_Initialise: [global] Initialise Time variables
        Time::TemporalSpacings: [singlemap] Set timestep based on Courant condition (courant_static)
        WeylScal4::WeylScal4_CheckBoundaries: [meta] check boundaries treatment
      [CCTK_RECOVER_VARIABLES]
        IOUtil::IOUtil_RecoverGH: [level] Checkpoint recovery routine
      [CCTK_POST_RECOVER_VARIABLES]
        CarpetIOHDF5::CarpetIOHDF5_InitCheckpointingIntervals: [global] Initialisation of checkpointing intervals after recovery
        CarpetIOHDF5::CarpetIOHDF5_CloseFiles: [meta] Close all initial data checkpoint files after recovery
        GROUP MaskBase_SetupMask: Set up the weight function
          GROUP MaskBase_SetupMaskAll: Set up the weight function
            CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
            CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
            GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
              CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
              CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
            GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
              CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
              CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
            CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
            GROUP SetupMask: Set up the real weight function (schedule other routines in here)
            CarpetReduce::MaskBase_TestMask: [global] Test the weight function
        GRHydro::GRHydro_EOSHandle: [global] Set the EOS number
        GRHydro::GRHydroCopyIntegerMask: Initialize the real valued atmosphere mask after checkpoint recovery
        GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
        GROUP HydroBase_ExcisionMaskSetup: Set up hydro excision mask
          HydroBase::HydroBase_InitExcisionMask: Initialize hydro excision mask to 'no excision everywhere'
        GROUP zzz_NaNChecker_NaNCheck: Check for NaNs and count them in NaNChecker::NaNsFound
          NaNChecker::NaNChecker_NaNCheck_Prepare: [level] Prepare data structures to check for NaNs
          NaNChecker::NaNChecker_NaNCheck_Check: [local] Check for NaNs
          NaNChecker::NaNChecker_NaNCheck_Finish: [level] Count NaNs in NaNChecker::NaNsFound
        NaNChecker::NaNChecker_TakeAction: [global] [loop-level] Output NaNChecker::NaNmask and take action according to NaNChecker::action_if_found
        TerminationTrigger::TerminationTrigger_ResetMinutes: [global] Reset Watchtime
    endif
    if (checkpoint initial data)
      [CCTK_CPINITIAL]
        CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint: [meta] Initial data checkpoint routine
    endif
    if (analysis)
      [CCTK_ANALYSIS]
        CarpetLib::CarpetLib_printtimestats: [global] Print timing statistics if desired
        CarpetLib::CarpetLib_printmemstats: [global] Print memory statistics if desired
        Formaline::Formaline_AnnounceUpdate: [global] Put some meta information about the current run into permanent storage
        LoopControl::LC_statistics_analysis: [meta] Output LoopControl statistics
        GROUP ML_BSSN_EvolutionAnalysis: Calculate RHS at analysis
          ML_BSSN::ML_BSSN_EvolutionAnalysisInit: ML_BSSN_EvolutionAnalysisInit
          ML_BSSN::ML_BSSN_EvolutionAnalysisInterior: ML_BSSN_EvolutionAnalysisInterior
          ML_BSSN_Helper::ML_BSSN_NewRad: Apply NewRad boundary conditions to RHS
        Multipole::Multipole_Calc: [global] Calculate Multipoles
        QuasiLocalMeasures::qlm_calculate: [global] Calculate quasi-local quantities
        SystemStatistics::SystemStatistics_Collect: [global] Collect system statistics
        TerminationTrigger::TerminationTrigger_CheckWalltime: Check elapsed job walltime
        TerminationTrigger::TerminationTrigger_CheckSignal: Check if we received a termination signal
        TerminationTrigger::TerminationTrigger_CheckFile: Check termination file
        TimerReport::zzz_TimerReport_Output: [global] Print the timer report
        Trigger::Trigger_Check: [global] test
  endif
  Output grid variables

  do loop over timesteps
    [CCTK_PREREGRID]
      CarpetTracker::CarpetTracker_SetPositions: [global] Set positions of refined regions
    Change grid hierarchy
    [CCTK_POSTREGRID]
      CartGrid3D::SpatialCoordinates: Set Coordinates after regridding
      GROUP MaskBase_SetupMask: Set up the weight function
        GROUP MaskBase_SetupMaskAll: Set up the weight function
          CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
          CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
          GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
            CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
            CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
          GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
            CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
            CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
          CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
          GROUP SetupMask: Set up the real weight function (schedule other routines in here)
          CarpetReduce::MaskBase_TestMask: [global] Test the weight function
      Dissipation::setup_epsdis: Setup spatially varying dissipation
      SpaceMask::MaskZero: Initialise mask to zero
      GROUP GRHydro_PrimitiveBoundaries: Apply boundary conditions to all primitives
        GRHydro::GRHydro_SelectPrimitiveBoundaries: [level] Select primitive variables for boundary conditions
        GROUP GRHydro_ApplyPrimitiveBCs: Apply boundary conditions to all primitive variables
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
      GROUP HydroBase_ExcisionMaskSetup: Set up hydro excision mask
        HydroBase::HydroBase_InitExcisionMask: Initialize hydro excision mask to 'no excision everywhere'
      SpaceMask::MaskOne: Set mask to one
      GRHydro::GRHydro_SetupMask: Initialize the atmosphere mask
      GROUP MoL_PostStep: Ensure that everything is correct after regridding
        ML_BSSN::ML_BSSN_SelectBoundConds: [level] select boundary conditions
        GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
        GRHydro::GRHydro_SetLastMoLPostStep: [level] Set grid scalar InLastMoLPostStep if this is the last MoL PostStep call
        GROUP ML_BSSN_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        ML_ADMConstraints::ML_ADMConstraints_SelectBoundConds: [level] select boundary conditions
        GROUP ML_ADMConstraints_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        ML_BSSN::ML_BSSN_ADMBaseInterior: ML_BSSN_ADMBaseInterior
        ML_BSSN::ML_BSSN_ADMBaseBoundaryScalar: ML_BSSN_ADMBaseBoundaryScalar
        ML_BSSN::ML_BSSN_ADMBaseEverywhere: ML_BSSN_ADMBaseEverywhere
        ML_BSSN_Helper::ML_BSSN_ADMBase_SelectBCs: [level] Select boundary conditions for ADMBase variables
        GROUP ML_BSSN_ADMBase_ApplyBCs: Apply boundary conditions to ADMBase variables
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
        GROUP HydroBase_PostStep: Post step tasks for hydro thorns
          GROUP GRHydro_PostStep: Post step tasks for GRHydro
          GROUP GRHydro_AtmosphereMaskBoundaries: Apply boundary conditions to primitives
            GRHydro::GRHydro_SelectAtmosphereMaskBoundaries: [level] Select atmosphere mask for boundary conditions
            GROUP GRHydro_ApplyAtmosphereMaskBCs: Apply boundary conditions to real-valued atmosphere mask
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GRHydro::GRHydroPostSyncAtmosphereMask: Set integer atmosphere mask from synchronized real atmosphere mask
          if (GRHydro::InLastMoLPostStep)
            GRHydro::GRHydro_AtmosphereReset: Reset the atmosphere
          if (GRHydro::InLastMoLPostStep)
            GROUP GRHydro_PrimitiveInitialGuessesBoundaries: Apply boundary conditions to those primitives used as initial guesses
              GRHydro::GRHydro_SelectPrimitiveInitialGuessesBoundaries: [level] Select initial guess primitive variables for boudary conditions
              GROUP GRHydro_ApplyPrimitiveInitialGuessBCs: Apply boundary conditions to initial guess primitive variables
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP HydroBase_Boundaries: HydroBase-internal Boundary conditions group
            GROUP Do_GRHydro_Boundaries: GRHydro Boundary conditions group
            GROUP HydroBase_Select_Boundaries: Group to schedule the boundary condition functions
              if (GRHydro::execute_MoL_PostStep)
                GRHydro::GRHydro_Bound: [level] Select GRHydro boundary conditions
            GROUP HydroBase_ApplyBCs: Apply the boundary conditions of HydroBase
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP HydroBase_Con2Prim: Convert from conservative to primitive variables
            if (GRHydro::execute_MoL_Step)
              GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
            if (GRHydro::execute_MoL_PostStep)
              GRHydro::Con2Prim: Convert back to primitive variables (general)
        GROUP SetTmunu: Group for calculating the stress-energy tensor
          TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
          GROUP AddToTmunu: Add to the stress-energy tensor here
            GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
        WeylScal4::WeylScal4_SelectBoundConds: [level] select boundary conditions
        GROUP WeylScal4_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
      GROUP MoL_PseudoEvolutionBoundaries: Apply boundary conditions to pseudo-evolved quantities
        GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
          ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
          GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
          ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
          GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
          ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
          GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
          WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
          GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
    Rotate timelevels
    iteration = iteration+1
    t = t+dt
    [CCTK_PRESTEP]
      CoordGauge::Einstein_SetNextSlicing: Identify the slicing for the next iteration
      GRHydro::reset_GRHydro_C2P_failed: Reset the mask function that contains the points where C2P has failed (at PRESTEP)
      LoopControl::LC_steer: [meta] Update LoopControl algorithm preferences
      NaNChecker::NaNChecker_ResetCounter: [global] Reset the NaNChecker::NaNsFound counter
    [CCTK_EVOL]
      MoL::MoL_StartLoop: [level] Initialise the step size control
      while (MoL::MoL_Stepsize_Bad)
        GROUP MoL_Evolution: A single Cactus evolution step using MoL
          GROUP MoL_StartStep: MoL internal setup for the evolution step
            MoL::MoL_SetCounter: [level] Set the counter for the ODE method to loop over
            MoL::MoL_SetTime: [level] Ensure the correct time and timestep are used
            MoL::MoL_AllocateScratchSpace: [level] Allocate storage for scratch levels
          GROUP MoL_PreStep: Physics thorns can schedule preloop setup routines in here
            GRHydro::GRHydro_Scalar_Setup: Set up and check scalars for efficiency
          MoL::MoL_AllocateScratch: Allocate sufficient space for array scratch variables
          MoL::MoL_InitialCopy: Ensure the data is in the correct timelevel
          while (MoL::MoL_Intermediate_Step)
            GROUP MoL_Step: The loop over the intermediate steps for the ODE integrator
              MoL::MoL_InitRHS: Initialise the RHS functions
              GROUP MoL_CalcRHS: Physics thorns schedule the calculation of the discrete spatial operator in here
                GROUP HydroBase_RHS: Groups for scheduling tasks for calculating RHS of hydro variables
                  if (GRHydro::execute_MoL_Step)
                    GROUP GRHydroRHS: Calculate the update terms
                      GRHydro::SourceTerms: Source term calculation
                      GRHydro::GRHydroStartLoop: [level] Set the flux_direction variable
                      while (GRHydro::flux_direction)
                        GROUP FluxTerms: Calculation of intercell fluxes
                          GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
                          GRHydro::Reconstruct: Reconstruct the functions at the cell boundaries
                          GRHydro::Riemann: Solve the local Riemann problems
                          GRHydro::UpdateCalcul: Calculate the update term from the fluxes
                          GRHydro::GRHydroAdvanceLoop: [level] Decrement the flux_direction variable
                      end while
                      GRHydro::GRHydroUpdateAtmosphereMask: Alter the update terms if inside the atmosphere region
                ML_BSSN::ML_BSSN_EvolutionBoundaryScalar: ML_BSSN_EvolutionBoundaryScalar
                GROUP ML_BSSN_EvolutionInteriorSplitBy: 
                  ML_BSSN::ML_BSSN_EvolutionInteriorSplitBy1: ML_BSSN_EvolutionInteriorSplitBy1
                  ML_BSSN::ML_BSSN_EvolutionInteriorSplitBy2: ML_BSSN_EvolutionInteriorSplitBy2
                  ML_BSSN::ML_BSSN_EvolutionInteriorSplitBy3: ML_BSSN_EvolutionInteriorSplitBy3
                ML_BSSN_Helper::ML_BSSN_NewRad: Apply NewRad boundary conditions to RHS
              GROUP MoL_PostRHS: Modify RHS functions
                Dissipation::dissipation_add: Add Kreiss-Oliger dissipation to the right hand sides
              GROUP MoL_RHSBoundaries: Any 'final' modifications to the RHS functions (boundaries etc.)
              MoL::MoL_Add: Updates calculated with the efficient Runge-Kutta 4 method
              MoL::MoL_DecrementCounter: [level] Alter the counter number
              MoL::MoL_ResetTime: [level] If necessary, change the time
              GROUP MoL_PostStepModify: The group for physics thorns to schedule enforcing constraints
                ML_BSSN::ML_BSSN_EnforceEverywhere: ML_BSSN_EnforceEverywhere
              GROUP MoL_PostStep: The group for physics thorns to schedule boundary calls etc.
                ML_BSSN::ML_BSSN_SelectBoundConds: [level] select boundary conditions
                GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
                GRHydro::GRHydro_SetLastMoLPostStep: [level] Set grid scalar InLastMoLPostStep if this is the last MoL PostStep call
                GROUP ML_BSSN_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
                  GROUP BoundaryConditions: Execute all boundary conditions
                    Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                    CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                    ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                    RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                    RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                  Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
                ML_ADMConstraints::ML_ADMConstraints_SelectBoundConds: [level] select boundary conditions
                GROUP ML_ADMConstraints_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
                  GROUP BoundaryConditions: Execute all boundary conditions
                    Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                    CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                    ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                    RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                    RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                  Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
                ML_BSSN::ML_BSSN_ADMBaseInterior: ML_BSSN_ADMBaseInterior
                ML_BSSN::ML_BSSN_ADMBaseBoundaryScalar: ML_BSSN_ADMBaseBoundaryScalar
                ML_BSSN::ML_BSSN_ADMBaseEverywhere: ML_BSSN_ADMBaseEverywhere
                ML_BSSN_Helper::ML_BSSN_ADMBase_SelectBCs: [level] Select boundary conditions for ADMBase variables
                GROUP ML_BSSN_ADMBase_ApplyBCs: Apply boundary conditions to ADMBase variables
                  GROUP BoundaryConditions: Execute all boundary conditions
                    Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                    CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                    ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                    RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                    RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                  Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
                GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
                GROUP HydroBase_PostStep: Post step tasks for hydro thorns
                  GROUP GRHydro_PostStep: Post step tasks for GRHydro
                  GROUP GRHydro_AtmosphereMaskBoundaries: Apply boundary conditions to primitives
                    GRHydro::GRHydro_SelectAtmosphereMaskBoundaries: [level] Select atmosphere mask for boundary conditions
                    GROUP GRHydro_ApplyAtmosphereMaskBCs: Apply boundary conditions to real-valued atmosphere mask
                      GROUP BoundaryConditions: Execute all boundary conditions
                        Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                        CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                        ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                        RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                        RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                      Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
                  GRHydro::GRHydroPostSyncAtmosphereMask: Set integer atmosphere mask from synchronized real atmosphere mask
                  if (GRHydro::InLastMoLPostStep)
                    GRHydro::GRHydro_AtmosphereReset: Reset the atmosphere
                  if (GRHydro::InLastMoLPostStep)
                    GROUP GRHydro_PrimitiveInitialGuessesBoundaries: Apply boundary conditions to those primitives used as initial guesses
                      GRHydro::GRHydro_SelectPrimitiveInitialGuessesBoundaries: [level] Select initial guess primitive variables for boudary conditions
                      GROUP GRHydro_ApplyPrimitiveInitialGuessBCs: Apply boundary conditions to initial guess primitive variables
                        GROUP BoundaryConditions: Execute all boundary conditions
                          Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                          CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                          ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                          RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                          RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                        Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
                  GROUP HydroBase_Boundaries: HydroBase-internal Boundary conditions group
                    GROUP Do_GRHydro_Boundaries: GRHydro Boundary conditions group
                    GROUP HydroBase_Select_Boundaries: Group to schedule the boundary condition functions
                      if (GRHydro::execute_MoL_PostStep)
                        GRHydro::GRHydro_Bound: [level] Select GRHydro boundary conditions
                    GROUP HydroBase_ApplyBCs: Apply the boundary conditions of HydroBase
                      GROUP BoundaryConditions: Execute all boundary conditions
                        Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                        CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                        ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                        RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                        RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                      Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
                  GROUP HydroBase_Con2Prim: Convert from conservative to primitive variables
                    if (GRHydro::execute_MoL_Step)
                      GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
                    if (GRHydro::execute_MoL_PostStep)
                      GRHydro::Con2Prim: Convert back to primitive variables (general)
                GROUP SetTmunu: Group for calculating the stress-energy tensor
                  TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
                  GROUP AddToTmunu: Add to the stress-energy tensor here
                    GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
                WeylScal4::WeylScal4_SelectBoundConds: [level] select boundary conditions
                GROUP WeylScal4_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
                  GROUP BoundaryConditions: Execute all boundary conditions
                    Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                    CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                    ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                    RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                    RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                  Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
              GRHydro::GRHydro_ClearLastMoLPostStep: [level] Reset InLastMoLPostStep to zero
              MoL::MoL_ResetDeltaTime: [level] If necessary, change the timestep
          end while
          MoL::MoL_FinishLoop: [level] Control the step size
          MoL::MoL_RestoreSandR: Restoring the Save and Restore variables to the original state
          MoL::MoL_FreeScratchSpace: [level] Free storage for scratch levels
      end while
      GRHydro::sync_GRHydro_C2P_failed: Syncronise the mask function that contains the points where C2P has failed
      GROUP MoL_PseudoEvolution: Calculate pseudo-evolved quantities
        GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
        GROUP ML_ADMConstraints_evaluate_group: ML_ADMConstraints_evaluate
          ML_ADMConstraints::ML_ADMConstraints_evaluate: ML_ADMConstraints_evaluate
          GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
            ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
            GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsEverywhere_group: ML_BSSN_ConstraintsEverywhere
          ML_BSSN::ML_BSSN_ConstraintsEverywhere: ML_BSSN_ConstraintsEverywhere
          GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
            ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
            GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsInterior_group: ML_BSSN_ConstraintsInterior
          ML_BSSN::ML_BSSN_ConstraintsInterior: ML_BSSN_ConstraintsInterior
          GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
            ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
            GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP WeylScal4_psi4_calc_4th_group: WeylScal4_psi4_calc_4th
          WeylScal4::WeylScal4_psi4_calc_4th: WeylScal4_psi4_calc_4th
          GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
            WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
            GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
    Evolve finer grids recursively
    Restrict from finer grids
    [CCTK_POSTRESTRICT]
      GROUP MoL_PostStep: Ensure that everything is correct after restriction
        ML_BSSN::ML_BSSN_SelectBoundConds: [level] select boundary conditions
        GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
        GRHydro::GRHydro_SetLastMoLPostStep: [level] Set grid scalar InLastMoLPostStep if this is the last MoL PostStep call
        GROUP ML_BSSN_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        ML_ADMConstraints::ML_ADMConstraints_SelectBoundConds: [level] select boundary conditions
        GROUP ML_ADMConstraints_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        ML_BSSN::ML_BSSN_ADMBaseInterior: ML_BSSN_ADMBaseInterior
        ML_BSSN::ML_BSSN_ADMBaseBoundaryScalar: ML_BSSN_ADMBaseBoundaryScalar
        ML_BSSN::ML_BSSN_ADMBaseEverywhere: ML_BSSN_ADMBaseEverywhere
        ML_BSSN_Helper::ML_BSSN_ADMBase_SelectBCs: [level] Select boundary conditions for ADMBase variables
        GROUP ML_BSSN_ADMBase_ApplyBCs: Apply boundary conditions to ADMBase variables
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
        GROUP HydroBase_PostStep: Post step tasks for hydro thorns
          GROUP GRHydro_PostStep: Post step tasks for GRHydro
          GROUP GRHydro_AtmosphereMaskBoundaries: Apply boundary conditions to primitives
            GRHydro::GRHydro_SelectAtmosphereMaskBoundaries: [level] Select atmosphere mask for boundary conditions
            GROUP GRHydro_ApplyAtmosphereMaskBCs: Apply boundary conditions to real-valued atmosphere mask
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GRHydro::GRHydroPostSyncAtmosphereMask: Set integer atmosphere mask from synchronized real atmosphere mask
          if (GRHydro::InLastMoLPostStep)
            GRHydro::GRHydro_AtmosphereReset: Reset the atmosphere
          if (GRHydro::InLastMoLPostStep)
            GROUP GRHydro_PrimitiveInitialGuessesBoundaries: Apply boundary conditions to those primitives used as initial guesses
              GRHydro::GRHydro_SelectPrimitiveInitialGuessesBoundaries: [level] Select initial guess primitive variables for boudary conditions
              GROUP GRHydro_ApplyPrimitiveInitialGuessBCs: Apply boundary conditions to initial guess primitive variables
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP HydroBase_Boundaries: HydroBase-internal Boundary conditions group
            GROUP Do_GRHydro_Boundaries: GRHydro Boundary conditions group
            GROUP HydroBase_Select_Boundaries: Group to schedule the boundary condition functions
              if (GRHydro::execute_MoL_PostStep)
                GRHydro::GRHydro_Bound: [level] Select GRHydro boundary conditions
            GROUP HydroBase_ApplyBCs: Apply the boundary conditions of HydroBase
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP HydroBase_Con2Prim: Convert from conservative to primitive variables
            if (GRHydro::execute_MoL_Step)
              GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
            if (GRHydro::execute_MoL_PostStep)
              GRHydro::Con2Prim: Convert back to primitive variables (general)
        GROUP SetTmunu: Group for calculating the stress-energy tensor
          TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
          GROUP AddToTmunu: Add to the stress-energy tensor here
            GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
        WeylScal4::WeylScal4_SelectBoundConds: [level] select boundary conditions
        GROUP WeylScal4_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
      GROUP MoL_PseudoEvolutionBoundaries: Apply boundary conditions to pseudo-evolved quantities
        GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
          ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
          GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
          ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
          GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
          ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
          GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
          WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
          GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
    [CCTK_POSTSTEP]
      SphericalSurface::SphericalSurface_Set: [global] Set surface radii
      GROUP SphericalSurface_HasBeenSet: Set the spherical surfaces before this group, and use it afterwards
        SphericalSurface::SphericalSurface_CheckState: [global] Test the state of the spherical surfaces
      GRHydro::GRHydro_RefinementLevel: Calculate current refinement level (for the check of the C2P mask)
      GRHydro::check_GRHydro_C2P_failed: Check the mask function that contains the points where C2P has failed and report an error in case a failure is found
      HTTPD::HTTP_Work: [global] Working routine
      HTTPD::HTTP_ContentWork: [global] Content Working routine
      GROUP HydroBase_ExcisionHasBeenSet: Group to schedule thorns changing the mask before and thorns using the mask after
      GROUP Hydro_Analysis: Group for Hydro_Analysis routines
        Hydro_Analysis::Hydro_Analysis_PrepareReduction: [global] [loop-local] Compute the local reduction results
        Hydro_Analysis::Hydro_Analysis_Reduction: [global] Compute the global reduction results
        GROUP Hydro_Analysis_LocationSearch: Look for the location of the maximum density
          Hydro_Analysis::Hydro_Analysis_LocationSearch_Setup: [global] Prepare data structures for search
          Hydro_Analysis::Hydro_Analysis_LocationSearch_Search: [global] [loop-local] Search for the location of the maximum density
          Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine: [global] Communicate and verify the location of the maximum density
        Hydro_Analysis::Hydro_Analysis_FindSeparation: [global] Compute the proper distance between the maximum of the density and the origin (along a straight coordinate line)
        NSTracker::NSTracker: [global] Look for the star and set grid accordingly
      GROUP zzz_NaNChecker_NaNCheck: Check for NaNs and count them in NaNChecker::NaNsFound
        NaNChecker::NaNChecker_NaNCheck_Prepare: [level] Prepare data structures to check for NaNs
        NaNChecker::NaNChecker_NaNCheck_Check: [local] Check for NaNs
        NaNChecker::NaNChecker_NaNCheck_Finish: [level] Count NaNs in NaNChecker::NaNsFound
      NaNChecker::NaNChecker_TakeAction: [global] [loop-level] Output NaNChecker::NaNmask and take action according to NaNChecker::action_if_found
      SpaceMask::CheckMask: Ensure that all mask values are legal
      Dissipation::setup_epsdis: Setup spatially varying dissipation
      GROUP MaskBase_SetupMask: Set up the weight function
        GROUP MaskBase_SetupMaskAll: Set up the weight function
          CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
          CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
          GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
            CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
            CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
          GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
            CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
            CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
          CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
          GROUP SetupMask: Set up the real weight function (schedule other routines in here)
          CarpetReduce::MaskBase_TestMask: [global] Test the weight function
    if (checkpoint)
      [CCTK_CHECKPOINT]
      CarpetIOHDF5::CarpetIOHDF5_EvolutionCheckpoint: [meta] Evolution checkpoint routine
      TimerReport::zzz_TimerReport_Checkpoint: [global] Print the timer report
    endif
    if (analysis)
      [CCTK_ANALYSIS]
      CarpetLib::CarpetLib_printtimestats: [global] Print timing statistics if desired
      CarpetLib::CarpetLib_printmemstats: [global] Print memory statistics if desired
      Formaline::Formaline_AnnounceUpdate: [global] Put some meta information about the current run into permanent storage
      LoopControl::LC_statistics_analysis: [meta] Output LoopControl statistics
      GROUP ML_BSSN_EvolutionAnalysis: Calculate RHS at analysis
        ML_BSSN::ML_BSSN_EvolutionAnalysisInit: ML_BSSN_EvolutionAnalysisInit
        ML_BSSN::ML_BSSN_EvolutionAnalysisInterior: ML_BSSN_EvolutionAnalysisInterior
        ML_BSSN_Helper::ML_BSSN_NewRad: Apply NewRad boundary conditions to RHS
      Multipole::Multipole_Calc: [global] Calculate Multipoles
      QuasiLocalMeasures::qlm_calculate: [global] Calculate quasi-local quantities
      SystemStatistics::SystemStatistics_Collect: [global] Collect system statistics
      TerminationTrigger::TerminationTrigger_CheckWalltime: Check elapsed job walltime
      TerminationTrigger::TerminationTrigger_CheckSignal: Check if we received a termination signal
      TerminationTrigger::TerminationTrigger_CheckFile: Check termination file
      TimerReport::zzz_TimerReport_Output: [global] Print the timer report
      Trigger::Trigger_Check: [global] test
    endif
    Output grid variables
    enddo

  Termination routines
    [CCTK_TERMINATE]
      CarpetIOHDF5::CarpetIOHDF5_TerminationCheckpoint: [meta] Termination checkpoint routine
      Formaline::Formaline_AnnounceFinal: [global] Put some meta information about the current run into permanent storage
      LoopControl::LC_statistics_terminate: [meta] Output LoopControl statistics
      MoL::MoL_FreeIndexArrays: Free the MoL bookkeeping index arrays
      TimerReport::zzz_TimerReport_Output: [global] Print the timer report

  Shutdown routines
    [CCTK_SHUTDOWN]
      HTTPD::HTTP_Shutdown: [global] HTTP daemon shutdown
      Timers::Timer_Shutdown: Prepare hierarchical timers

  Routines run after changing the grid hierarchy:
    [CCTK_POSTREGRID]
      CartGrid3D::SpatialCoordinates: Set Coordinates after regridding
      GROUP MaskBase_SetupMask: Set up the weight function
        GROUP MaskBase_SetupMaskAll: Set up the weight function
          CarpetReduce::MaskBase_AllocateMask: [global] Allocate the weight function
          CarpetReduce::MaskBase_InitMask: [global] [loop-local] Initialise the weight function
          GROUP SetupIMaskInternal: Set up the integer weight function (schedule other routines in here)
            CarpetReduce::CoordBase_SetupMask: [global] [loop-local] Set up the outer boundaries of the weight function
            CarpetReduce::CarpetMaskSetup: [global] [loop-singlemap] Set up the weight function for the restriction regions
          GROUP SetupIMask: Set up the integer weight function (schedule other routines in here)
            CarpetMask::CarpetExcludedSetup: [global] [loop-local] Set up the weight function for the excluded regions
            CarpetMask::CarpetSurfaceSetup: [global] [loop-local] Set up the weight function for the excluded spherical surfaces
          CarpetReduce::MaskBase_SetMask: [global] [loop-local] Set the weight function
          GROUP SetupMask: Set up the real weight function (schedule other routines in here)
          CarpetReduce::MaskBase_TestMask: [global] Test the weight function
      Dissipation::setup_epsdis: Setup spatially varying dissipation
      SpaceMask::MaskZero: Initialise mask to zero
      GROUP GRHydro_PrimitiveBoundaries: Apply boundary conditions to all primitives
        GRHydro::GRHydro_SelectPrimitiveBoundaries: [level] Select primitive variables for boundary conditions
        GROUP GRHydro_ApplyPrimitiveBCs: Apply boundary conditions to all primitive variables
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
      GROUP HydroBase_ExcisionMaskSetup: Set up hydro excision mask
        HydroBase::HydroBase_InitExcisionMask: Initialize hydro excision mask to 'no excision everywhere'
      SpaceMask::MaskOne: Set mask to one
      GRHydro::GRHydro_SetupMask: Initialize the atmosphere mask
      GROUP MoL_PostStep: Ensure that everything is correct after regridding
        ML_BSSN::ML_BSSN_SelectBoundConds: [level] select boundary conditions
        GRHydro::GRHydro_RefinementLevel: Calculate current refinement level
        GRHydro::GRHydro_SetLastMoLPostStep: [level] Set grid scalar InLastMoLPostStep if this is the last MoL PostStep call
        GROUP ML_BSSN_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        ML_ADMConstraints::ML_ADMConstraints_SelectBoundConds: [level] select boundary conditions
        GROUP ML_ADMConstraints_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        ML_BSSN::ML_BSSN_ADMBaseInterior: ML_BSSN_ADMBaseInterior
        ML_BSSN::ML_BSSN_ADMBaseBoundaryScalar: ML_BSSN_ADMBaseBoundaryScalar
        ML_BSSN::ML_BSSN_ADMBaseEverywhere: ML_BSSN_ADMBaseEverywhere
        ML_BSSN_Helper::ML_BSSN_ADMBase_SelectBCs: [level] Select boundary conditions for ADMBase variables
        GROUP ML_BSSN_ADMBase_ApplyBCs: Apply boundary conditions to ADMBase variables
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ADMBase_SetADMVars: Set the ADM variables before this group, and use them afterwards
        GROUP HydroBase_PostStep: Post step tasks for hydro thorns
          GROUP GRHydro_PostStep: Post step tasks for GRHydro
          GROUP GRHydro_AtmosphereMaskBoundaries: Apply boundary conditions to primitives
            GRHydro::GRHydro_SelectAtmosphereMaskBoundaries: [level] Select atmosphere mask for boundary conditions
            GROUP GRHydro_ApplyAtmosphereMaskBCs: Apply boundary conditions to real-valued atmosphere mask
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GRHydro::GRHydroPostSyncAtmosphereMask: Set integer atmosphere mask from synchronized real atmosphere mask
          if (GRHydro::InLastMoLPostStep)
            GRHydro::GRHydro_AtmosphereReset: Reset the atmosphere
          if (GRHydro::InLastMoLPostStep)
            GROUP GRHydro_PrimitiveInitialGuessesBoundaries: Apply boundary conditions to those primitives used as initial guesses
              GRHydro::GRHydro_SelectPrimitiveInitialGuessesBoundaries: [level] Select initial guess primitive variables for boudary conditions
              GROUP GRHydro_ApplyPrimitiveInitialGuessBCs: Apply boundary conditions to initial guess primitive variables
                GROUP BoundaryConditions: Execute all boundary conditions
                  Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                  CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                  ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                  RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                  RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
                Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP HydroBase_Boundaries: HydroBase-internal Boundary conditions group
            GROUP Do_GRHydro_Boundaries: GRHydro Boundary conditions group
            GROUP HydroBase_Select_Boundaries: Group to schedule the boundary condition functions
              if (GRHydro::execute_MoL_PostStep)
                GRHydro::GRHydro_Bound: [level] Select GRHydro boundary conditions
            GROUP HydroBase_ApplyBCs: Apply the boundary conditions of HydroBase
              GROUP BoundaryConditions: Execute all boundary conditions
                Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
                CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
                ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
                RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
                RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
              Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
          GROUP HydroBase_Con2Prim: Convert from conservative to primitive variables
            if (GRHydro::execute_MoL_Step)
              GRHydro::GRHydro_SqrtSpatialDeterminant: Calculate sdetg
            if (GRHydro::execute_MoL_PostStep)
              GRHydro::Con2Prim: Convert back to primitive variables (general)
        GROUP SetTmunu: Group for calculating the stress-energy tensor
          TmunuBase::TmunuBase_ZeroTmunu: Initialise the stress-energy tensor to zero
          GROUP AddToTmunu: Add to the stress-energy tensor here
            GRHydro::GRHydro_Tmunu: Compute the energy-momentum tensor
        WeylScal4::WeylScal4_SelectBoundConds: [level] select boundary conditions
        GROUP WeylScal4_ApplyBCs: Apply boundary conditions controlled by thorn Boundary
          GROUP BoundaryConditions: Execute all boundary conditions
            Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
            CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
            ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
            RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
            RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
          Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
      GROUP MoL_PseudoEvolutionBoundaries: Apply boundary conditions to pseudo-evolved quantities
        GROUP ML_ADMConstraints_evaluate_bc_group: ML_ADMConstraints_evaluate
          ML_ADMConstraints::ML_ADMConstraints_evaluate_SelectBCs: [level] ML_ADMConstraints_evaluate_SelectBCs
          GROUP ML_ADMConstraints_evaluate_ApplyBCs: Apply BCs for groups set in ML_ADMConstraints_evaluate
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsEverywhere_bc_group: ML_BSSN_ConstraintsEverywhere
          ML_BSSN::ML_BSSN_ConstraintsEverywhere_SelectBCs: [level] ML_BSSN_ConstraintsEverywhere_SelectBCs
          GROUP ML_BSSN_ConstraintsEverywhere_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsEverywhere
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP ML_BSSN_ConstraintsInterior_bc_group: ML_BSSN_ConstraintsInterior
          ML_BSSN::ML_BSSN_ConstraintsInterior_SelectBCs: [level] ML_BSSN_ConstraintsInterior_SelectBCs
          GROUP ML_BSSN_ConstraintsInterior_ApplyBCs: Apply BCs for groups set in ML_BSSN_ConstraintsInterior
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
        GROUP WeylScal4_psi4_calc_4th_bc_group: WeylScal4_psi4_calc_4th
          WeylScal4::WeylScal4_psi4_calc_4th_SelectBCs: [level] WeylScal4_psi4_calc_4th_SelectBCs
          GROUP WeylScal4_psi4_calc_4th_ApplyBCs: Apply BCs for groups set in WeylScal4_psi4_calc_4th
            GROUP BoundaryConditions: Execute all boundary conditions
              Boundary::Boundary_ApplyPhysicalBCs: Apply all requested local physical boundary conditions
              CartGrid3D::CartGrid3D_ApplyBC: Apply symmetry boundary conditions
              ReflectionSymmetry::ReflectionSymmetry_Apply: Apply reflection symmetries
              RotatingSymmetry180::Rot180_ComputeLevelExtent: [level] Compute extent of level
              RotatingSymmetry180::Rot180_ApplyBC: Apply 180 degree rotational symmetry boundary condition
            Boundary::Boundary_ClearSelection: [level] Unselect all grid variables for boundary conditions
--------------------------------------------------------------------------------
INFO (Carpet): Multi-Model listing:
   model 0: "world"
INFO (Carpet): Multi-Model process distribution:
   process 0: model 0 "world"
INFO (Carpet): Multi-Model: This is process 0, model 0 "world"
INFO (CarpetLib): Process startup time was 1.06 seconds
Current core file size limit: hard=[unlimited], soft=[unlimited]
Current addres space size limit: hard=[unlimited], soft=[unlimited]
Current data segment size limit: hard=[unlimited], soft=[unlimited]
Current resident set size limit: hard=[unlimited], soft=[unlimited]
INFO (CycleClock): Measuring CycleClock tick via OpenMP...
INFO (CycleClock): Calibrated CycleClock: 0.34549 ns per clock tick (2.89444 GHz)
INFO (hwloc): library version 2.0.4, API version 0x20000
INFO (SystemTopology): MPI process-to-host mapping:
This is MPI process 0 of 1
MPI hosts:
  0: panther
This MPI process runs on host 0 of 1
On this host, this is MPI process 0 of 1
INFO (SystemTopology): Topology support:
Discovery support:
  discovery->pu                            : yes
CPU binding support:
  cpubind->set_thisproc_cpubind            : yes
  cpubind->get_thisproc_cpubind            : yes
  cpubind->set_proc_cpubind                : yes
  cpubind->get_proc_cpubind                : yes
  cpubind->set_thisthread_cpubind          : yes
  cpubind->get_thisthread_cpubind          : yes
  cpubind->set_thread_cpubind              : yes
  cpubind->get_thread_cpubind              : yes
  cpubind->get_thisproc_last_cpu_location  : yes
  cpubind->get_proc_last_cpu_location      : yes
  cpubind->get_thisthread_last_cpu_location: yes
Memory binding support:
  membind->set_thisproc_membind            : no
  membind->get_thisproc_membind            : no
  membind->set_proc_membind                : no
  membind->get_proc_membind                : no
  membind->set_thisthread_membind          : yes
  membind->get_thisthread_membind          : yes
  membind->set_area_membind                : yes
  membind->get_area_membind                : yes
  membind->alloc_membind                   : yes
  membind->firsttouch_membind              : yes
  membind->bind_membind                    : yes
  membind->interleave_membind              : yes
  membind->nexttouch_membind               : no
  membind->migrate_membind                 : yes
INFO (SystemTopology): Hardware objects in this node:
Machine L#0: (P#0, total=1043114796KB, DMIProductName="PowerEdge R7515", DMIProductVersion=, DMIBoardVendor="Dell Inc.", DMIBoardName=04F3CJ, DMIBoardVersion=A02, DMIChassisVendor="Dell Inc.", DMIChassisType=23, DMIChassisVersion=, DMIChassisAssetTag=, DMIBIOSVendor="Dell Inc.", DMIBIOSVersion=2.18.1, DMIBIOSDate=02/03/2025, DMISysVendor="Dell Inc.", Backend=Linux, OSName=Linux, OSRelease=5.14.0-570.37.1.el9_6.x86_64, OSVersion="#1 SMP PREEMPT_DYNAMIC Thu Aug 28 10:41:06 UTC 2025", HostName=panther, Architecture=x86_64, hwlocVersion=2.0.4, ProcessName=cactus_sim)
  Package L#0: (P#0, total=1043114796KB, CPUVendor=AuthenticAMD, CPUFamilyNumber=23, CPUModelNumber=49, CPUModel="AMD EPYC 7542 32-Core Processor                ", CPUStepping=0)
    L3Cache L#0: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#0: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#0: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#0: (P#0)
            PU L#0: (P#0)
            PU L#1: (P#32)
      L2Cache L#1: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#1: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#1: (P#1)
            PU L#2: (P#1)
            PU L#3: (P#33)
      L2Cache L#2: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#2: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#2: (P#2)
            PU L#4: (P#2)
            PU L#5: (P#34)
      L2Cache L#3: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#3: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#3: (P#3)
            PU L#6: (P#3)
            PU L#7: (P#35)
    L3Cache L#1: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#4: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#4: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#4: (P#4)
            PU L#8: (P#4)
            PU L#9: (P#36)
      L2Cache L#5: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#5: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#5: (P#5)
            PU L#10: (P#5)
            PU L#11: (P#37)
      L2Cache L#6: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#6: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#6: (P#6)
            PU L#12: (P#6)
            PU L#13: (P#38)
      L2Cache L#7: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#7: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#7: (P#7)
            PU L#14: (P#7)
            PU L#15: (P#39)
    L3Cache L#2: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#8: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#8: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#8: (P#8)
            PU L#16: (P#8)
            PU L#17: (P#40)
      L2Cache L#9: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#9: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#9: (P#9)
            PU L#18: (P#9)
            PU L#19: (P#41)
      L2Cache L#10: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#10: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#10: (P#10)
            PU L#20: (P#10)
            PU L#21: (P#42)
      L2Cache L#11: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#11: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#11: (P#11)
            PU L#22: (P#11)
            PU L#23: (P#43)
    L3Cache L#3: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#12: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#12: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#12: (P#12)
            PU L#24: (P#12)
            PU L#25: (P#44)
      L2Cache L#13: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#13: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#13: (P#13)
            PU L#26: (P#13)
            PU L#27: (P#45)
      L2Cache L#14: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#14: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#14: (P#14)
            PU L#28: (P#14)
            PU L#29: (P#46)
      L2Cache L#15: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#15: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#15: (P#15)
            PU L#30: (P#15)
            PU L#31: (P#47)
    L3Cache L#4: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#16: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#16: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#16: (P#16)
            PU L#32: (P#16)
            PU L#33: (P#48)
      L2Cache L#17: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#17: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#17: (P#17)
            PU L#34: (P#17)
            PU L#35: (P#49)
      L2Cache L#18: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#18: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#18: (P#18)
            PU L#36: (P#18)
            PU L#37: (P#50)
      L2Cache L#19: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#19: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#19: (P#19)
            PU L#38: (P#19)
            PU L#39: (P#51)
    L3Cache L#5: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#20: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#20: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#20: (P#20)
            PU L#40: (P#20)
            PU L#41: (P#52)
      L2Cache L#21: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#21: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#21: (P#21)
            PU L#42: (P#21)
            PU L#43: (P#53)
      L2Cache L#22: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#22: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#22: (P#22)
            PU L#44: (P#22)
            PU L#45: (P#54)
      L2Cache L#23: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#23: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#23: (P#23)
            PU L#46: (P#23)
            PU L#47: (P#55)
    L3Cache L#6: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#24: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#24: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#24: (P#24)
            PU L#48: (P#24)
            PU L#49: (P#56)
      L2Cache L#25: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#25: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#25: (P#25)
            PU L#50: (P#25)
            PU L#51: (P#57)
      L2Cache L#26: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#26: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#26: (P#26)
            PU L#52: (P#26)
            PU L#53: (P#58)
      L2Cache L#27: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#27: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#27: (P#27)
            PU L#54: (P#27)
            PU L#55: (P#59)
    L3Cache L#7: (P#-1, size=16384KB, linesize=64, ways=16, Inclusive=0)
      L2Cache L#28: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#28: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#28: (P#28)
            PU L#56: (P#28)
            PU L#57: (P#60)
      L2Cache L#29: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#29: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#29: (P#29)
            PU L#58: (P#29)
            PU L#59: (P#61)
      L2Cache L#30: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#30: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#30: (P#30)
            PU L#60: (P#30)
            PU L#61: (P#62)
      L2Cache L#31: (P#-1, size=512KB, linesize=64, ways=8, Inclusive=1)
        L1dCache L#31: (P#-1, size=32KB, linesize=64, ways=8, Inclusive=0)
          Core L#31: (P#31)
            PU L#62: (P#31)
            PU L#63: (P#63)
INFO (SystemTopology): Thread CPU bindings:
  MPI process 0 on host 0 (process 0 of 1 on this host)
    OpenMP thread 0: PU set L#{0-63} P#{0-63}
INFO (SystemTopology): Setting thread CPU bindings:
INFO (SystemTopology): Thread CPU bindings:
  MPI process 0 on host 0 (process 0 of 1 on this host)
    OpenMP thread 0: PU set L#{0} P#{0}
INFO (SystemTopology): Extracting CPU/cache/memory properties:
  There are 2 PUs per core (aka hardware SMT threads)
  There are 1 threads per core (aka SMT threads used)
  Cache (unknown name) has type "data" depth 1
    size 32768 linesize 64 associativity 8 stride 4096, for 2 PUs
  Cache (unknown name) has type "unified" depth 2
    size 524288 linesize 64 associativity 8 stride 65536, for 2 PUs
  Cache (unknown name) has type "unified" depth 3
    size 16777216 linesize 64 associativity 16 stride 1048576, for 8 PUs
INFO (Formaline): Configuration id: config-sim-panther-system-user-crangano-einstein_toolkit-BNS
INFO (Formaline): Build id: build-sim-panther-crangano-2025.09.15-19.43.32-3940620
INFO (Formaline): Simulation id: run-bns-panther-crangano-2025.09.15-20.32.02-3948578
INFO (Formaline): Run id: run-bns-panther-crangano-2025.09.15-20.32.02-3948578
Server started on http://panther:5555/
Not announcing location via Twitter.
INFO (Vectors): Using vector size 2 for architecture SSE2+SSE4.1 (64-bit precision)
--------------------------------------------------------------------------------
AMR driver provided by Carpet
--------------------------------------------------------------------------------
GRHydro: relativistic hydrodynamics, no ice.
--------------------------------------------------------------------------------
HydroBase: Let it flow.
--------------------------------------------------------------------------------
AMR info I/O provided by CarpetIOBasic
--------------------------------------------------------------------------------
ML_ADMConstraints
--------------------------------------------------------------------------------
ML_BSSN
--------------------------------------------------------------------------------
AMR HDF5 I/O provided by CarpetIOHDF5
--------------------------------------------------------------------------------
AMR 0D HDF5 I/O provided by CarpetIOHDF5
--------------------------------------------------------------------------------
AMR 1D HDF5 I/O provided by CarpetIOHDF5
--------------------------------------------------------------------------------
AMR 2D HDF5 I/O provided by CarpetIOHDF5
--------------------------------------------------------------------------------
AMR 3D HDF5 I/O provided by CarpetIOHDF5
--------------------------------------------------------------------------------
MoL: Generalized time integration.
--------------------------------------------------------------------------------
AMR scalar I/O provided by CarpetIOScalar
--------------------------------------------------------------------------------
AMR 0D ASCII I/O provided by CarpetIOASCII
--------------------------------------------------------------------------------
AMR 1D ASCII I/O provided by CarpetIOASCII
--------------------------------------------------------------------------------
AMR 2D ASCII I/O provided by CarpetIOASCII
--------------------------------------------------------------------------------
AMR 3D ASCII I/O provided by CarpetIOASCII
--------------------------------------------------------------------------------
WeylScal4
--------------------------------------------------------------------------------

INFO (Carpet): MPI is enabled
INFO (Carpet): Carpet is running on 1 processes
INFO (Carpet): This is process 0
INFO (Carpet): OpenMP is enabled
INFO (Carpet): This process contains 1 threads, this is thread 0
INFO (Carpet): There are 1 threads in total
INFO (Carpet): There are 1 threads per process
INFO (Carpet): This process runs on host panther, pid=3948578
INFO (Carpet): This process runs on 1 core: 0
INFO (Carpet): Thread 0 runs on 1 core: 0
INFO (Carpet): This simulation is running in 3 dimensions
INFO (Carpet): Boundary specification for map 0:
   nboundaryzones: [[3,3,3],[3,3,3]]
   is_internal   : [[0,0,0],[0,0,0]]
   is_staggered  : [[0,0,0],[0,0,0]]
   shiftout      : [[1,0,1],[0,0,0]]
INFO (Carpet): CoordBase domain specification for map 0:
   physical extent: [0,-400,0] : [400,400,400]   ([400,800,400])
   interior extent: [0,-392,0] : [392,392,392]   ([392,784,392])
   exterior extent: [-24,-416,-24] : [416,416,416]   ([440,832,440])
   base_spacing   : [8,8,8]
INFO (Carpet): Adapted domain specification for map 0:
   convergence factor: 2
   convergence level : 0
   physical extent   : [0,-400,0] : [400,400,400]   ([400,800,400])
   interior extent   : [0,-392,0] : [392,392,392]   ([392,784,392])
   exterior extent   : [-24,-416,-24] : [416,416,416]   ([440,832,440])
   spacing           : [8,8,8]
INFO (Carpet): Base grid specification for map 0:
   number of grid points             : [56,105,56]
   number of coarse grid ghost points: [[3,3,3],[3,3,3]]
INFO (Carpet): Buffer zone counts (excluding ghosts):
   [0]: [[0,0,0],[0,0,0]]
   [1]: [[9,9,9],[9,9,9]]
   [2]: [[9,9,9],[9,9,9]]
   [3]: [[9,9,9],[9,9,9]]
   [4]: [[9,9,9],[9,9,9]]
   [5]: [[9,9,9],[9,9,9]]
   [6]: [[9,9,9],[9,9,9]]
   [7]: [[9,9,9],[9,9,9]]
   [8]: [[9,9,9],[9,9,9]]
INFO (Carpet): Overlap zone counts:
   [0]: [[0,0,0],[0,0,0]]
   [1]: [[0,0,0],[0,0,0]]
   [2]: [[0,0,0],[0,0,0]]
   [3]: [[0,0,0],[0,0,0]]
   [4]: [[0,0,0],[0,0,0]]
   [5]: [[0,0,0],[0,0,0]]
   [6]: [[0,0,0],[0,0,0]]
   [7]: [[0,0,0],[0,0,0]]
   [8]: [[0,0,0],[0,0,0]]
INFO (Carpet): Group and variable statistics:
INFO (Carpet):    There are 1162 grid functions in 159 groups
INFO (Carpet):    There are 604 grid scalars in 96 groups
INFO (Carpet):    There are 130 1-dimensional grid arrays in 13 groups
INFO (Carpet):    There are 143 2-dimensional grid arrays in 18 groups
INFO (Carpet):    There are 0 3-dimensional grid arrays in 0 groups
INFO (Carpet):    (The number of variables counts all time levels)
INFO (CarpetIOHDF5): I/O Method 'IOHDF5' registered: AMR output of grid variables to HDF5 files
INFO (CarpetIOHDF5): I/O Method 'IOHDF5_0D' registered: 0D AMR output of grid variables to HDF5 files
INFO (CarpetIOHDF5): I/O Method 'IOHDF5_1D' registered: 1D AMR output of grid variables to HDF5 files
INFO (CarpetIOHDF5): I/O Method 'IOHDF5_2D' registered: 2D AMR output of grid variables to HDF5 files
INFO (CarpetIOHDF5): Periodic 2D AMR output requested for:
   ADMBASE::gxx
   ADMBASE::gxy
   ADMBASE::gxz
   ADMBASE::gyy
   ADMBASE::gyz
   ADMBASE::gzz
   ADMBASE::alp
   ADMBASE::betax
   ADMBASE::betay
   ADMBASE::betaz
   CARPETREDUCE::weight
   GRID::x
   GRID::y
   GRID::z
   GRID::r
   HYDROBASE::rho
   HYDROBASE::eps
   HYDROBASE::vel[0]
   HYDROBASE::vel[1]
   HYDROBASE::vel[2]
   ML_ADMCONSTRAINTS::H
INFO (CarpetIOHDF5): I/O Method 'IOHDF5_3D' registered: 3D AMR output of grid variables to HDF5 files
INFO (CarpetIOHDF5): Periodic 3D AMR output requested for:
   ADMBASE::alp
   ADMBASE::betax
   ADMBASE::betay
   ADMBASE::betaz
   CARPETREDUCE::weight
   GRID::x
   GRID::y
   GRID::z
   GRID::r
   HYDROBASE::rho
   HYDROBASE::eps
   HYDROBASE::vel[0]
   HYDROBASE::vel[1]
   HYDROBASE::vel[2]
   ML_ADMCONSTRAINTS::H
INFO (CarpetIOScalar): Periodic scalar output requested for:
   ADMBASE::gxx
   ADMBASE::gxy
   ADMBASE::gxz
   ADMBASE::gyy
   ADMBASE::gyz
   ADMBASE::gzz
   ADMBASE::kxx
   ADMBASE::kxy
   ADMBASE::kxz
   ADMBASE::kyy
   ADMBASE::kyz
   ADMBASE::kzz
   ADMBASE::alp
   ADMBASE::betax
   ADMBASE::betay
   ADMBASE::betaz
   GRHYDRO::dens
   HYDROBASE::rho
   HYDROBASE::vel[0]
   HYDROBASE::vel[1]
   HYDROBASE::vel[2]
   HYDROBASE::w_lorentz
   ML_ADMCONSTRAINTS::H
   SPHERICALSURFACE::sf_radius[0]
   SPHERICALSURFACE::sf_radius[1]
   SPHERICALSURFACE::sf_radius[2]
   SPHERICALSURFACE::sf_radius[3]
   SPHERICALSURFACE::sf_radius[4]
   SYSTEMSTATISTICS::maxrss_mb
   SYSTEMSTATISTICS::majflt_mb
   SYSTEMSTATISTICS::arena_mb
   SYSTEMSTATISTICS::ordblks_mb
   SYSTEMSTATISTICS::hblks_mb
   SYSTEMSTATISTICS::hblkhd_mb
   SYSTEMSTATISTICS::uordblks_mb
   SYSTEMSTATISTICS::fordblks_mb
   SYSTEMSTATISTICS::keepcost_mb
   SYSTEMSTATISTICS::swap_used_mb
INFO (CarpetIOASCII): I/O Method 'IOASCII_0D' registered: 0D AMR output of grid variables to ASCII files
INFO (CarpetIOASCII): Periodic 0D AMR output requested for:
   CARPET::physical_time_per_hour
   CARPET::current_physical_time_per_hour
   CARPET::time_total
   CARPET::time_evolution
   CARPET::time_computing
   CARPET::time_communicating
   CARPET::time_io
   CARPET::evolution_steps_count
   CARPET::local_grid_points_per_second
   CARPET::total_grid_points_per_second
   CARPET::local_grid_point_updates_count
   CARPET::total_grid_point_updates_count
   CARPET::local_interior_points_per_second
   CARPET::total_interior_points_per_second
   CARPET::local_interior_point_updates_count
   CARPET::total_interior_point_updates_count
   CARPET::io_per_second
   CARPET::io_bytes_per_second
   CARPET::io_bytes_ascii_per_second
   CARPET::io_bytes_binary_per_second
   CARPET::io_count
   CARPET::io_bytes_count
   CARPET::io_bytes_ascii_count
   CARPET::io_bytes_binary_count
   CARPET::comm_per_second
   CARPET::comm_bytes_per_second
   CARPET::comm_count
   CARPET::comm_bytes_count
   CARPET::time_levels
   CARPET::current_walltime
   CARPET::syncs_count
   HYDRO_ANALYSIS::Hydro_Analysis_rho_max_loc[0]
   HYDRO_ANALYSIS::Hydro_Analysis_rho_max_loc[1]
   HYDRO_ANALYSIS::Hydro_Analysis_rho_max_loc[2]
   HYDRO_ANALYSIS::Hydro_Analysis_rho_max_origin_distance
   QUASILOCALMEASURES::qlm_time[0]
   QUASILOCALMEASURES::qlm_time[1]
   QUASILOCALMEASURES::qlm_equatorial_circumference[0]
   QUASILOCALMEASURES::qlm_equatorial_circumference[1]
   QUASILOCALMEASURES::qlm_polar_circumference_0[0]
   QUASILOCALMEASURES::qlm_polar_circumference_0[1]
   QUASILOCALMEASURES::qlm_polar_circumference_pi_2[0]
   QUASILOCALMEASURES::qlm_polar_circumference_pi_2[1]
   QUASILOCALMEASURES::qlm_area[0]
   QUASILOCALMEASURES::qlm_area[1]
   QUASILOCALMEASURES::qlm_irreducible_mass[0]
   QUASILOCALMEASURES::qlm_irreducible_mass[1]
   QUASILOCALMEASURES::qlm_radius[0]
   QUASILOCALMEASURES::qlm_radius[1]
   QUASILOCALMEASURES::qlm_spin_guess[0]
   QUASILOCALMEASURES::qlm_spin_guess[1]
   QUASILOCALMEASURES::qlm_mass_guess[0]
   QUASILOCALMEASURES::qlm_mass_guess[1]
   QUASILOCALMEASURES::qlm_killing_eigenvalue_re[0]
   QUASILOCALMEASURES::qlm_killing_eigenvalue_re[1]
   QUASILOCALMEASURES::qlm_killing_eigenvalue_im[0]
   QUASILOCALMEASURES::qlm_killing_eigenvalue_im[1]
   QUASILOCALMEASURES::qlm_spin[0]
   QUASILOCALMEASURES::qlm_spin[1]
   QUASILOCALMEASURES::qlm_npspin[0]
   QUASILOCALMEASURES::qlm_npspin[1]
   QUASILOCALMEASURES::qlm_wsspin[0]
   QUASILOCALMEASURES::qlm_wsspin[1]
   QUASILOCALMEASURES::qlm_cvspin[0]
   QUASILOCALMEASURES::qlm_cvspin[1]
   QUASILOCALMEASURES::qlm_coordspinx[0]
   QUASILOCALMEASURES::qlm_coordspinx[1]
   QUASILOCALMEASURES::qlm_coordspiny[0]
   QUASILOCALMEASURES::qlm_coordspiny[1]
   QUASILOCALMEASURES::qlm_coordspinz[0]
   QUASILOCALMEASURES::qlm_coordspinz[1]
   QUASILOCALMEASURES::qlm_mass[0]
   QUASILOCALMEASURES::qlm_mass[1]
   QUASILOCALMEASURES::qlm_adm_energy[0]
   QUASILOCALMEASURES::qlm_adm_energy[1]
   QUASILOCALMEASURES::qlm_adm_momentum_x[0]
   QUASILOCALMEASURES::qlm_adm_momentum_x[1]
   QUASILOCALMEASURES::qlm_adm_momentum_y[0]
   QUASILOCALMEASURES::qlm_adm_momentum_y[1]
   QUASILOCALMEASURES::qlm_adm_momentum_z[0]
   QUASILOCALMEASURES::qlm_adm_momentum_z[1]
   QUASILOCALMEASURES::qlm_adm_angular_momentum_x[0]
   QUASILOCALMEASURES::qlm_adm_angular_momentum_x[1]
   QUASILOCALMEASURES::qlm_adm_angular_momentum_y[0]
   QUASILOCALMEASURES::qlm_adm_angular_momentum_y[1]
   QUASILOCALMEASURES::qlm_adm_angular_momentum_z[0]
   QUASILOCALMEASURES::qlm_adm_angular_momentum_z[1]
   QUASILOCALMEASURES::qlm_w_energy[0]
   QUASILOCALMEASURES::qlm_w_energy[1]
   QUASILOCALMEASURES::qlm_w_momentum_x[0]
   QUASILOCALMEASURES::qlm_w_momentum_x[1]
   QUASILOCALMEASURES::qlm_w_momentum_y[0]
   QUASILOCALMEASURES::qlm_w_momentum_y[1]
   QUASILOCALMEASURES::qlm_w_momentum_z[0]
   QUASILOCALMEASURES::qlm_w_momentum_z[1]
   QUASILOCALMEASURES::qlm_w_angular_momentum_x[0]
   QUASILOCALMEASURES::qlm_w_angular_momentum_x[1]
   QUASILOCALMEASURES::qlm_w_angular_momentum_y[0]
   QUASILOCALMEASURES::qlm_w_angular_momentum_y[1]
   QUASILOCALMEASURES::qlm_w_angular_momentum_z[0]
   QUASILOCALMEASURES::qlm_w_angular_momentum_z[1]
   SPHERICALSURFACE::sf_active[0]
   SPHERICALSURFACE::sf_active[1]
   SPHERICALSURFACE::sf_active[2]
   SPHERICALSURFACE::sf_active[3]
   SPHERICALSURFACE::sf_active[4]
   SPHERICALSURFACE::sf_valid[0]
   SPHERICALSURFACE::sf_valid[1]
   SPHERICALSURFACE::sf_valid[2]
   SPHERICALSURFACE::sf_valid[3]
   SPHERICALSURFACE::sf_valid[4]
   SPHERICALSURFACE::sf_area[0]
   SPHERICALSURFACE::sf_area[1]
   SPHERICALSURFACE::sf_area[2]
   SPHERICALSURFACE::sf_area[3]
   SPHERICALSURFACE::sf_area[4]
   SPHERICALSURFACE::sf_mean_radius[0]
   SPHERICALSURFACE::sf_mean_radius[1]
   SPHERICALSURFACE::sf_mean_radius[2]
   SPHERICALSURFACE::sf_mean_radius[3]
   SPHERICALSURFACE::sf_mean_radius[4]
   SPHERICALSURFACE::sf_centroid_x[0]
   SPHERICALSURFACE::sf_centroid_x[1]
   SPHERICALSURFACE::sf_centroid_x[2]
   SPHERICALSURFACE::sf_centroid_x[3]
   SPHERICALSURFACE::sf_centroid_x[4]
   SPHERICALSURFACE::sf_centroid_y[0]
   SPHERICALSURFACE::sf_centroid_y[1]
   SPHERICALSURFACE::sf_centroid_y[2]
   SPHERICALSURFACE::sf_centroid_y[3]
   SPHERICALSURFACE::sf_centroid_y[4]
   SPHERICALSURFACE::sf_centroid_z[0]
   SPHERICALSURFACE::sf_centroid_z[1]
   SPHERICALSURFACE::sf_centroid_z[2]
   SPHERICALSURFACE::sf_centroid_z[3]
   SPHERICALSURFACE::sf_centroid_z[4]
   SPHERICALSURFACE::sf_quadrupole_xx[0]
   SPHERICALSURFACE::sf_quadrupole_xx[1]
   SPHERICALSURFACE::sf_quadrupole_xx[2]
   SPHERICALSURFACE::sf_quadrupole_xx[3]
   SPHERICALSURFACE::sf_quadrupole_xx[4]
   SPHERICALSURFACE::sf_quadrupole_xy[0]
   SPHERICALSURFACE::sf_quadrupole_xy[1]
   SPHERICALSURFACE::sf_quadrupole_xy[2]
   SPHERICALSURFACE::sf_quadrupole_xy[3]
   SPHERICALSURFACE::sf_quadrupole_xy[4]
   SPHERICALSURFACE::sf_quadrupole_xz[0]
   SPHERICALSURFACE::sf_quadrupole_xz[1]
   SPHERICALSURFACE::sf_quadrupole_xz[2]
   SPHERICALSURFACE::sf_quadrupole_xz[3]
   SPHERICALSURFACE::sf_quadrupole_xz[4]
   SPHERICALSURFACE::sf_quadrupole_yy[0]
   SPHERICALSURFACE::sf_quadrupole_yy[1]
   SPHERICALSURFACE::sf_quadrupole_yy[2]
   SPHERICALSURFACE::sf_quadrupole_yy[3]
   SPHERICALSURFACE::sf_quadrupole_yy[4]
   SPHERICALSURFACE::sf_quadrupole_yz[0]
   SPHERICALSURFACE::sf_quadrupole_yz[1]
   SPHERICALSURFACE::sf_quadrupole_yz[2]
   SPHERICALSURFACE::sf_quadrupole_yz[3]
   SPHERICALSURFACE::sf_quadrupole_yz[4]
   SPHERICALSURFACE::sf_quadrupole_zz[0]
   SPHERICALSURFACE::sf_quadrupole_zz[1]
   SPHERICALSURFACE::sf_quadrupole_zz[2]
   SPHERICALSURFACE::sf_quadrupole_zz[3]
   SPHERICALSURFACE::sf_quadrupole_zz[4]
   SPHERICALSURFACE::sf_min_radius[0]
   SPHERICALSURFACE::sf_min_radius[1]
   SPHERICALSURFACE::sf_min_radius[2]
   SPHERICALSURFACE::sf_min_radius[3]
   SPHERICALSURFACE::sf_min_radius[4]
   SPHERICALSURFACE::sf_max_radius[0]
   SPHERICALSURFACE::sf_max_radius[1]
   SPHERICALSURFACE::sf_max_radius[2]
   SPHERICALSURFACE::sf_max_radius[3]
   SPHERICALSURFACE::sf_max_radius[4]
   SPHERICALSURFACE::sf_min_x[0]
   SPHERICALSURFACE::sf_min_x[1]
   SPHERICALSURFACE::sf_min_x[2]
   SPHERICALSURFACE::sf_min_x[3]
   SPHERICALSURFACE::sf_min_x[4]
   SPHERICALSURFACE::sf_min_y[0]
   SPHERICALSURFACE::sf_min_y[1]
   SPHERICALSURFACE::sf_min_y[2]
   SPHERICALSURFACE::sf_min_y[3]
   SPHERICALSURFACE::sf_min_y[4]
   SPHERICALSURFACE::sf_min_z[0]
   SPHERICALSURFACE::sf_min_z[1]
   SPHERICALSURFACE::sf_min_z[2]
   SPHERICALSURFACE::sf_min_z[3]
   SPHERICALSURFACE::sf_min_z[4]
   SPHERICALSURFACE::sf_max_x[0]
   SPHERICALSURFACE::sf_max_x[1]
   SPHERICALSURFACE::sf_max_x[2]
   SPHERICALSURFACE::sf_max_x[3]
   SPHERICALSURFACE::sf_max_x[4]
   SPHERICALSURFACE::sf_max_y[0]
   SPHERICALSURFACE::sf_max_y[1]
   SPHERICALSURFACE::sf_max_y[2]
   SPHERICALSURFACE::sf_max_y[3]
   SPHERICALSURFACE::sf_max_y[4]
   SPHERICALSURFACE::sf_max_z[0]
   SPHERICALSURFACE::sf_max_z[1]
   SPHERICALSURFACE::sf_max_z[2]
   SPHERICALSURFACE::sf_max_z[3]
   SPHERICALSURFACE::sf_max_z[4]
   SPHERICALSURFACE::sf_radius[0]
   SPHERICALSURFACE::sf_radius[1]
   SPHERICALSURFACE::sf_radius[2]
   SPHERICALSURFACE::sf_radius[3]
   SPHERICALSURFACE::sf_radius[4]
   SPHERICALSURFACE::sf_origin_x[0]
   SPHERICALSURFACE::sf_origin_x[1]
   SPHERICALSURFACE::sf_origin_x[2]
   SPHERICALSURFACE::sf_origin_x[3]
   SPHERICALSURFACE::sf_origin_x[4]
   SPHERICALSURFACE::sf_origin_y[0]
   SPHERICALSURFACE::sf_origin_y[1]
   SPHERICALSURFACE::sf_origin_y[2]
   SPHERICALSURFACE::sf_origin_y[3]
   SPHERICALSURFACE::sf_origin_y[4]
   SPHERICALSURFACE::sf_origin_z[0]
   SPHERICALSURFACE::sf_origin_z[1]
   SPHERICALSURFACE::sf_origin_z[2]
   SPHERICALSURFACE::sf_origin_z[3]
   SPHERICALSURFACE::sf_origin_z[4]
   SPHERICALSURFACE::sf_origin_theta[0]
   SPHERICALSURFACE::sf_origin_theta[1]
   SPHERICALSURFACE::sf_origin_theta[2]
   SPHERICALSURFACE::sf_origin_theta[3]
   SPHERICALSURFACE::sf_origin_theta[4]
   SPHERICALSURFACE::sf_origin_phi[0]
   SPHERICALSURFACE::sf_origin_phi[1]
   SPHERICALSURFACE::sf_origin_phi[2]
   SPHERICALSURFACE::sf_origin_phi[3]
   SPHERICALSURFACE::sf_origin_phi[4]
   SPHERICALSURFACE::sf_delta_theta[0]
   SPHERICALSURFACE::sf_delta_theta[1]
   SPHERICALSURFACE::sf_delta_theta[2]
   SPHERICALSURFACE::sf_delta_theta[3]
   SPHERICALSURFACE::sf_delta_theta[4]
   SPHERICALSURFACE::sf_delta_phi[0]
   SPHERICALSURFACE::sf_delta_phi[1]
   SPHERICALSURFACE::sf_delta_phi[2]
   SPHERICALSURFACE::sf_delta_phi[3]
   SPHERICALSURFACE::sf_delta_phi[4]
INFO (CarpetIOASCII): I/O Method 'IOASCII_1D' registered: 1D AMR output of grid variables to ASCII files
INFO (CarpetIOASCII): Periodic 1D AMR output requested for:
   ADMBASE::gxx
   ADMBASE::gxy
   ADMBASE::gxz
   ADMBASE::gyy
   ADMBASE::gyz
   ADMBASE::gzz
   ADMBASE::kxx
   ADMBASE::kxy
   ADMBASE::kxz
   ADMBASE::kyy
   ADMBASE::kyz
   ADMBASE::kzz
   ADMBASE::alp
   ADMBASE::betax
   ADMBASE::betay
   ADMBASE::betaz
   HYDROBASE::rho
   HYDROBASE::vel[0]
   HYDROBASE::vel[1]
   HYDROBASE::vel[2]
   ML_ADMCONSTRAINTS::H
INFO (CarpetIOASCII): I/O Method 'IOASCII_2D' registered: 2D AMR output of grid variables to ASCII files
INFO (CarpetIOASCII): I/O Method 'IOASCII_3D' registered: 3D AMR output of grid variables to ASCII files
--------------------------------------------------------------------------------
INFO (Formaline): Writing tarballs with the Cactus sources into the directory "bns/cactus-source"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:10:
  -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:25:
  -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:45:
  -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never"
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:114:
  -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:132:
  -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0
INFO (MoL): Using Runge-Kutta 4 as the time integrator.
INFO (SymBase): Symmetry on lower x-face: rotating_symmetry_180
INFO (SymBase): Symmetry on lower z-face: reflection_symmetry
INFO (MoL): The maximum number of evolved variables is 1246. 30 are registered.
INFO (MoL): The maximum number of slow evolved variables is 1246. 0 are registered.
INFO (MoL): The maximum number of constrained variables is 1246. 37 are registered.
INFO (MoL): The maximum number of SandR variables is 1246. 0 are registered.
INFO (MoL): The maximum number of evolved array variables is 1246. 0 are registered.
INFO (MoL): The maximum number of constrained array variables is 1246. 0 are registered.
INFO (MoL): The maximum number of SandR array variables is 1246. 0 are registered.
INFO (MoL): The maximum size of any array variables is 0.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:145:
  -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:152:
  -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:159:
  -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:166:
  -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:193:
  -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse.
WARNING level 1 from host panther process 0
  in thorn ML_BSSN_Helper, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/ML_BSSN_Helper/Parameters.c:199:
  -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift.
WARNING level 2 from host panther process 0
  in thorn TerminationTrigger, file /system/user/crangano/einstein_toolkit/BNS/configs/sim/build/TerminationTrigger/paramcheck.c:10:
  -> on_remaining_walltime (30 minutes) is more than max_walltime (0.333333 h) and will be reset to 0.0 to avoid an immediate termination
INFO (Vectors): Testing vectorisation... [errors may result in segfaults]
INFO (Vectors): 375/375 tests passed 
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
INFO (Carpet): Grid structure (superregions, grid points):
   [0][0][0]   exterior: [0,0,0] : [55,104,55]   ([56,105,56] + PADDING) 329280
   [1][0][0]   exterior: [3,35,3] : [79,173,75]   ([77,139,73] + PADDING) 781319
   [2][0][0]   exterior: [9,139,9] : [89,277,81]   ([81,139,73] + PADDING) 821907
   [3][0][0]   exterior: [21,347,21] : [109,485,93]   ([89,139,73] + PADDING) 903083
   [4][0][0]   exterior: [45,771,45] : [141,893,109]   ([97,123,65] + PADDING) 775515
   [5][0][0]   exterior: [93,1583,93] : [237,1745,177]   ([145,163,85] + PADDING) 2008975
   [6][0][0]   exterior: [189,3215,189] : [427,3441,305]   ([239,227,117] + PADDING) 6347601
INFO (Carpet): Grid structure (superregions, coordinates):
   [0][0][0]   exterior: [-24,-416,-24] : [416,416,416] : [8,8,8]
   [1][0][0]   exterior: [-12,-276,-12] : [292,276,276] : [4,4,4]
   [2][0][0]   exterior: [-6,-138,-6] : [154,138,138] : [2,2,2]
   [3][0][0]   exterior: [-3,-69,-3] : [85,69,69] : [1,1,1]
   [4][0][0]   exterior: [-1.5,-30.5,-1.5] : [46.5,30.5,30.5] : [0.5,0.5,0.5]
   [5][0][0]   exterior: [-0.75,-20.25,-0.75] : [35.25,20.25,20.25] : [0.25,0.25,0.25]
   [6][0][0]   exterior: [-0.375,-14.125,-0.375] : [29.375,14.125,14.125] : [0.125,0.125,0.125]
INFO (Carpet): Global grid structure statistics:
INFO (Carpet): GF: rhs: 2986k active, 3736k owned (+25%), 4184k total (+12%), 127 steps/time
INFO (Carpet): GF: vars: 319, pts: 2765M active, 3600M owned (+30%), 4145M total (+15%), 1.0 comp/proc
INFO (Carpet): GA: vars: 808, pts: 0M active, 0M total (+0%)
INFO (Carpet): Total required memory: 32.960 GByte (for GAs and currently active GFs)
INFO (Carpet): Load balance:  min     avg     max     sdv     max/avg-1
INFO (Carpet): Level  0:      79M     79M     79M      0M owned      0%
INFO (Carpet): Level  1:     230M    230M    230M      0M owned      0%
INFO (Carpet): Level  2:     242M    242M    242M      0M owned      0%
INFO (Carpet): Level  3:     267M    267M    267M      0M owned      0%
INFO (Carpet): Level  4:     229M    229M    229M      0M owned      0%
INFO (Carpet): Level  5:     605M    605M    605M      0M owned      0%
INFO (Carpet): Level  6:    1948M   1948M   1948M      0M owned      0%
INFO (CartGrid3D): Grid Spacings:
INFO (CartGrid3D): dx=>8.0000000e+00  dy=>8.0000000e+00  dz=>8.0000000e+00
INFO (CartGrid3D): Computational Coordinates:
INFO (CartGrid3D): x=>[-24.000,416.000]  y=>[-416.000,416.000]  z=>[-24.000,416.000]
INFO (CartGrid3D): Indices of Physical Coordinates:
INFO (CartGrid3D): x=>[0,55]  y=>[0,104]  z=>[0,55]
INFO (TerminationTrigger): Reminding you every 60 minutes about remaining walltime
INFO (Time): Timestep set to 3.2 (courant_static)
INFO (GRHydro): Trying to get EOS handles
INFO (GRHydro): Trying to get EOS handles
INFO (GRHydro): GRHydro will use the Ideal_Fluid equation of state.
INFO (GRHydro): Setting up the atmosphere mask: all points are not_atmosphere
INFO (Meudon_Bin_NS): Setting up LORENE Bin_NS initial data
INFO (Meudon_Bin_NS): Setting up coordinates
INFO (Meudon_Bin_NS): Meudon_Bin_NS will use the 2D_Polytrope equation of state.
INFO (Meudon_Bin_NS): Meudon_Bin_NS will use the 1 eos handle
INFO (Meudon_Bin_NS): Reading from file "/system/user/crangano/einstein_toolkit/BNS/bns.resu"
Simfactory Done at date: Mon 15 Sep 2025 10:32:12 PM CEST
-------------- next part --------------
#!/usr/bin/env python

# Copyright Barry Wardell, Ian Hinder, Eloisa Bentivegna

# We ask that if you make use of the parameter file or the example
# data, then please cite

# Simulation of GW150914 binary black hole merger using the
# Einstein Toolkit - https://doi.org/10.5281/zenodo.155394

# as well as the Einstein Toolkit, the Llama multi-block
# infrastructure, the Carpet mesh-refinement driver, the apparent
# horizon finder AHFinderDirect, the TwoPunctures initial data code,
# QuasiLocalMeasures, Cactus, and the McLachlan spacetime evolution
# code, the Kranc code generation package, and the Simulation Factory.

# An appropriate bibtex file, etgw150914.bib, is provided with this
# parameter file.

from math import *
import sys
import re
from string import Template

################################################################################
# Binary black hole configuration
################################################################################

# BHs labeled as '+' and '-' ('p' and 'm') for their initial position
# on the x axis.  The more massive BH is '+'.

D = 10.0                    # Separation
q = 36.0/29.0               # Mass ratio: q = mp/mm >= 1
M = 1.0                     # Total mass
chip =  0.31                # Dimensionsless z component of spin of + BH (x0 > 0, more massive)
chim = -0.46                # Dimensionsless z component of spin of - BH (x0 < 0, less massive)
Pr = -0.00084541526517121;  # Radial linear momentum
Pphi = 0.09530152296974252; # Azimuthal linear momentum

################################################################################
# Initial data
################################################################################

mp = M * q/(1+q)           # Heavier, larger BH, AH1, SS 0
mm = M * 1/(1+q)           # Lighter, smaller BH, AH2, SS 1

xp     = D * mm
xm     = -D * mp
half_D = D/2.0

center_offset = xp - half_D

Spx = 0.;
Spy = 0.;
Spz = chip * mp**2;

Smx = 0.;
Smy = 0.;
Smz = chim * mm**2;

Ppx = Pr
Ppy = Pphi
Ppz = 0

Pmx = -Ppx
Pmy = -Ppy
Pmz = -Ppz

################################################################################
# Grid structure
################################################################################

sphere_inner_radius = 45
expected_merger = 1000.0

# Number of cells across finest grid radius
n = int("2") if "2"[0] != "@" else 28
i = int(n/4)

n_angular = 2*i*2

# Estimated eventual AH radii
ahrp = mp * 1.0
ahrm = mm * 1.0

# Coordinate size of finest boxes around each BH
rp = ahrp * 1.2
rm = ahrm * 1.2

# Minimum acceptable radial grid spacing
hr_min = 2*0.96*M
maxrls = 9  # Essentially determines iteration counting

# Smaller '-' BH

# Decisions are made independent of n, to avoid roundoff problems.
# This is achieved by using nmin for the calculations and scaling by
# n/nmin at the end.

n_min = 24 # Cells across AHm radius
hfm_min = rm/n_min # Fine grid spacing for '-' BH
rlsm = 1 + int(log(hr_min/hfm_min,2)) # Number of refinements to attain hr_min
h0_min = hfm_min * 2**(rlsm-1) # RL0 Cartesian spacing

# Radii of each level for the centre around the '-' BH
levelsm = "[0," + ",".join(["%f" %(rm*2**(rlsm-l-1)) for l in range(1,rlsm)])+"]"

# '+' BH

# Number of levels for '+' BH estimated to keep error in angular
# velocity the same as for the '-' BH, assuming vErr \propto v *
# (h0/2**(rls-1)/rAH)**8
rlsp = rlsm + log(ahrm/ahrp * (xp/-xm)**(1./8))/log(2)
rlsp = int(round(rlsp))
levelsp = "[0," + ",".join(["%f" %(rp*2**(rlsp-l-1)) for l in range(1,rlsp)])+"]"

hr = h0_min * float(n_min)/float(n) # This could be changed (h0_min ->
                                    # hr_min) to get the minimum
                                    # radial resolution for efficiency
h0 = h0_min * float(n_min)/float(n)

time_after_merger = 200.0
waveform_length = expected_merger + time_after_merger
outermost_detector = 500.0
final_time = waveform_length + outermost_detector
sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
sphere_outer_radius = int(sphere_outer_radius / hr) * hr + hr # round up to a multiple of hr
sphere_inner_radius = int(ceil(sphere_inner_radius/(i*h0))) * h0 * i

################################################################################
# Frequencies
################################################################################

dtdx = 0.45 # Courant factor
dtdx0 = dtdx * 0.5 # RL0 is evolved with the same frequency as RL1
dt0 = dtdx0 * h0 # Time step on RL0
rl0_every = 2**(maxrls-2)
rl1_every = rl0_every

rl_max = maxrls-1
dt_it = dt0/2.0**(rl_max-1) # Time step of one iteration
find_cah = max(int((expected_merger - 200)/dt_it), 0)
wave_extract_every = rl0_every * 2 # Every other coarse grid step
                                   # (TODO: should this be every
                                   # coarse grid step?)
horizon_every = rl0_every * 2
out_every = rl0_every
# out3d_every = rl0_every * 2
out3d_every = 0
out2d_every = rl0_every * 8

################################################################################
# Sanity checks
################################################################################

# this example uses
# Coordinates::symmetry                   = "+z bitant"
# thus all data must by mirror symmetric accross the z=0 plane requiring 0
# values for z components of all polar vectors at the z=0 plane and 0 values
# for the x,y components of axial vectors

if (Spx != 0. or Spy != 0. or Smx != 0. or Smy != 0.):
    sys.stderr.write("This parameter file employs z-symmetry via the 'Coordinates::symmetry=\"+z bitant\"' setting. Therefore all spins must be in the z-direction only.\n");
    sys.exit(1)

if (Ppz != 0. or Pmz != 0.):
    sys.stderr.write("This parameter file employs z-symmetry via the 'Coordinates::symmetry=\"+z bitant\"' setting. Therefore no momentum may have a non-zero z-component.\n");
    sys.exit(1)

################################################################################
# Record all script variables in generated parameter file
################################################################################

local_vars = locals()

col_width = 0
for key,val in sorted(local_vars.items()):
    if isinstance(val, (int, float, complex, str)):
        col_width = max(len(str(key)), col_width)

var_settings = []
for key,val in sorted(local_vars.items()):
    if isinstance(val, (int, float, complex, str)):
        var_settings = var_settings + ["# {0}{1} = {2}".format(key," "*(col_width-len(key)),val)]

var_settings_str = "\n".join(var_settings)
        
lines = """
################################################################################
# Script variables
################################################################################

$var_settings_str

################################################################################
# Active thorns
################################################################################

ActiveThorns = "
  ADMBase
  ML_ADMConstraints
  ADMCoupling
  ADMMacros
  AEILocalInterp
  AHFinderDirect
  Boundary
  Carpet
  CarpetIOASCII
  CarpetIOBasic
  CarpetIOHDF5
  CarpetIOScalar
  CarpetInterp
  CarpetInterp2
  CarpetLib
  CarpetReduce
  CarpetRegrid2
  CarpetTracker
  CartGrid3D
  CoordBase
  CoordGauge
  Coordinates
  CoordinatesSymmetry
  Formaline
  GlobalDerivative
  hwloc
  IOUtil
  InitBase
  Interpolate2
  QuasiLocalMeasures
  LocalInterp
  LoopControl
  MoL
  NaNChecker
  PunctureTracker
  Slab
  SpaceMask
  SphericalSurface
  StaticConformal
  SummationByParts
  SymBase
  SystemStatistics
  SystemTopology
  TerminationTrigger
  TensorTypes
  Time
  TmunuBase
  TwoPunctures
  Vectors
  ML_BSSN
  ML_BSSN_Helper
  NewRad
  GenericFD
  WeylScal4
  Multipole
#  WaveExtractCPM
  ADMDerivatives
"

################################################################################
# Grid structure
################################################################################

Carpet::domain_from_multipatch          = yes
CartGrid3D::type                        = "multipatch"
CartGrid3D::set_coordinate_ranges_on    = "all maps"
Coordinates::coordinate_system          = "Thornburg04"
Coordinates::h_cartesian                = $h0
Coordinates::h_radial                   = $hr

Coordinates::sphere_inner_radius        = $sphere_inner_radius
Coordinates::sphere_outer_radius        = $sphere_outer_radius
Coordinates::n_angular                  = $n_angular

Driver::ghost_size                      = 5
Coordinates::patch_boundary_size        = 5
Coordinates::additional_overlap_size    = 3
Coordinates::outer_boundary_size        = 5

# change these (to their defaults) to disable z-zymmetry
CoordinatesSymmetry::reflection_z       = yes
CoordinatesSymmetry::stagger            = no
Coordinates::symmetry                   = "+z bitant"
Coordinates::additional_symmetry_size   = 1
Coordinates::verbose                    = no

Time::timestep_method                   = "given"
Time::timestep                          = $dt0
Carpet::time_refinement_factors         = "[1,1,2,4,8,16,32,64,128,256]"

################################################################################
# Mesh refinement
################################################################################

Carpet::max_refinement_levels           = $maxrls
CarpetRegrid2::num_centres              = 2
CarpetRegrid2::num_levels_1             = $rlsp
CarpetRegrid2::position_x_1             = $xp
CarpetRegrid2::radius_1                 = $levelsp
CarpetRegrid2::num_levels_2             = $rlsm
CarpetRegrid2::position_x_2             = $xm
CarpetRegrid2::radius_2                 = $levelsm

Carpet::use_buffer_zones                = yes
Carpet::prolongation_order_space        = 5
Carpet::prolongation_order_time         = 2

CarpetRegrid2::regrid_every             = $rl1_every
CarpetRegrid2::verbose                  = no
Carpet::grid_coordinates_filename       = "carpet-grid.asc"

################################################################################
# Initial data
################################################################################

ADMBase::initial_data                   = "twopunctures"
ADMBase::initial_lapse                  = "twopunctures-averaged"
ADMBase::initial_shift                  = "zero"
ADMBase::initial_dtlapse                = "zero"
ADMBase::initial_dtshift                = "zero"

TwoPunctures::target_M_plus             = $mp
TwoPunctures::target_M_minus            = $mm

# Initial guesses for iterative solve
TwoPunctures::par_m_plus                = $mp
TwoPunctures::par_m_minus               = $mm

TwoPunctures::par_b                     = $half_D
TwoPunctures::center_offset[0]          = $center_offset

TwoPunctures::par_P_plus[0]             = $Ppx
TwoPunctures::par_P_plus[1]             = $Ppy
TwoPunctures::par_P_plus[2]             = $Ppz

TwoPunctures::par_P_minus[0]            = $Pmx
TwoPunctures::par_P_minus[1]            = $Pmy
TwoPunctures::par_P_minus[2]            = $Pmz

TwoPunctures::par_S_plus[0]             = $Spx
TwoPunctures::par_S_plus[1]             = $Spy
TwoPunctures::par_S_plus[2]             = $Spz

TwoPunctures::par_S_minus[0]            = $Smx
TwoPunctures::par_S_minus[1]            = $Smy
TwoPunctures::par_S_minus[2]            = $Smz

TwoPunctures::grid_setup_method         = "evaluation"
TwoPunctures::give_bare_mass            = no
TwoPunctures::TP_epsilon                = 1e-6
Carpet::init_fill_timelevels            = yes
InitBase::initial_data_setup_method     = "init_single_level"

################################################################################
# Evolution and boundary
################################################################################

GenericFD::jacobian_group = "Coordinates::jacobian"
GenericFD::jacobian_derivative_group = "Coordinates::jacobian2"
GenericFD::jacobian_identity_map = 0

ADMBase::evolution_method         = "ML_BSSN"
ADMBase::lapse_evolution_method   = "ML_BSSN"
ADMBase::shift_evolution_method   = "ML_BSSN"
ADMBase::dtlapse_evolution_method = "ML_BSSN"
ADMBase::dtshift_evolution_method = "ML_BSSN"

ML_BSSN::fdOrder             = 8

# 1+log slicing requires harmonicN = 1 and harmonicF = 2.0
ML_BSSN::harmonicN           = 1
ML_BSSN::harmonicF           = 2.0

ML_BSSN::ShiftGammaCoeff     = 0.75
ML_BSSN::BetaDriver          = 1.0
ML_BSSN::advectLapse         = 1
ML_BSSN::advectShift         = 1

ML_BSSN::MinimumLapse        = 1.0e-8

# conformalaMethod = 1 for W, 0 for phi
ML_BSSN::conformalMethod     = 1

# We apply dissipation using GlobalDerivaitive so disable it here
ML_BSSN::epsDiss             = 0.0

ML_BSSN::dt_lapse_shift_method = "noLapseShiftAdvection"

ML_BSSN::initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::rhs_boundary_condition     = "scalar"
Boundary::radpower                  = 2

################################################################################
# BH tracking
################################################################################

CarpetTracker::surface                      [0] = 0
CarpetTracker::surface                      [1] = 1
PunctureTracker::track                      [0] = yes
PunctureTracker::initial_x                  [0] = $xp
PunctureTracker::which_surface_to_store_info[0] = 0
PunctureTracker::track                      [1] = yes
PunctureTracker::initial_x                  [1] = $xm
PunctureTracker::which_surface_to_store_info[1] = 1

################################################################################
# Spatial finite differencing
################################################################################

SummationByParts::order                              = 8

# Drop order instead of using upwinded stencils, only for advection derivatives
SummationByParts::sbp_upwind_deriv = no

SummationByParts::sbp_1st_deriv                      = yes
SummationByParts::sbp_2nd_deriv                      = no
SummationByParts::onesided_interpatch_boundaries     = no
SummationByParts::onesided_outer_boundaries          = yes
SummationByParts::use_dissipation                    = no
GlobalDerivative::use_dissipation                    = yes
SummationByParts::scale_with_h                       = yes
SummationByParts::dissipation_type                   = "Kreiss-Oliger"
SummationByParts::epsdis                             = 0.15

# Because dt/dx is half that of the other levels we also need half the dissipation
GlobalDerivative::epsdis_for_level               [0] = 0.075

# Variables for dissipation
SummationByParts::vars                               = "
  ML_BSSN::ML_log_confac
  ML_BSSN::ML_metric
  ML_BSSN::ML_trace_curv
  ML_BSSN::ML_curv
  ML_BSSN::ML_Gamma
  ML_BSSN::ML_lapse
  ML_BSSN::ML_shift
  ML_BSSN::ML_dtlapse
  ML_BSSN::ML_dtshift
"

################################################################################
# Time integration
################################################################################

MoL::ODE_Method                         = "rk4"
MoL::MoL_Intermediate_Steps             = 4
MoL::MoL_Num_Scratch_Levels             = 1

################################################################################
# Interpolation
################################################################################

CarpetInterp::check_tree_search         = no
CarpetInterp::tree_search               = yes
# Use 5-th order interpatch interpolation on the Llama grid
Interpolate::interpolator_order         = 5

################################################################################
# Psi4 computation
################################################################################

WeylScal4::fdOrder                   = 8
WeylScal4::calc_scalars              = "psis"
WeylScal4::calc_invariants           = "always"

################################################################################
# Psi4 mode decomposition
################################################################################

# Radii are chosen to be evenly spaced in 1/r as that is the variable
# extrapolation is performed in
Multipole::nradii       = 7
Multipole::radius[0]    = 100
Multipole::radius[1]    = 115
Multipole::radius[2]    = 136
Multipole::radius[3]    = 167
Multipole::radius[4]    = 214
Multipole::radius[5]    = 300
Multipole::radius[6]    = 500
Multipole::ntheta       = 120
Multipole::nphi         = 240
Multipole::variables    = "WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='psi4'}"
Multipole::out_every    = $wave_extract_every
Multipole::l_max        = 8
Multipole::output_hdf5  = yes

# Disable ASCII output to avoid creating a large number of files
Multipole::output_ascii = no

################################################################################
# Gauge invariant perturbations of Schwarzschild (ZM-CPM variables)
################################################################################

# WaveExtractCPM::out_every                  = $wave_extract_every
# WaveExtractCPM::use_carpetinterp2          = no
# WaveExtractCPM::calc_when_necessary        = no
# WaveExtractCPM::verbose                    = 0
# WaveExtractCPM::maximum_detector_number    = 7
# WaveExtractCPM::switch_output_format       = 100
# WaveExtractCPM::rsch2_computation          = "average Schwarzschild metric"
# WaveExtractCPM::l_mode                     = 8
# WaveExtractCPM::m_mode                     = 8
# WaveExtractCPM::detector_radius        [0] = 100
# WaveExtractCPM::detector_radius        [1] = 115
# WaveExtractCPM::detector_radius        [2] = 136
# WaveExtractCPM::detector_radius        [3] = 167
# WaveExtractCPM::detector_radius        [4] = 214
# WaveExtractCPM::detector_radius        [5] = 300
# WaveExtractCPM::detector_radius        [6] = 500
# WaveExtractCPM::maxntheta                  = 120
# WaveExtractCPM::maxnphi                    = 240
# WaveExtractCPM::output_hdf5                = yes
# WaveExtractCPM::output_ascii               = no
# WaveExtractCPM::output_h                   = yes
# WaveExtractCPM::output_Psi                 = yes

################################################################################
# Apparent Horizons
################################################################################

AHFinderDirect::N_horizons                               = 3
AHFinderDirect::find_every                               = $horizon_every
AHFinderDirect::output_h_every                           = 0
AHFinderDirect::max_Newton_iterations__initial           = 50
AHFinderDirect::max_Newton_iterations__subsequent        = 50
AHFinderDirect::max_allowable_Theta_growth_iterations    = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name               = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars               = "order=4"
AHFinderDirect::surface_interpolator_name                = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars                = "order=4"
AHFinderDirect::verbose_level                            = "physics details"
AHFinderDirect::move_origins                             = yes

AHFinderDirect::origin_x                             [1] = $xp
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = $xp
AHFinderDirect::initial_guess__coord_sphere__radius  [1] = $rp
AHFinderDirect::which_surface_to_store_info          [1] = 2
AHFinderDirect::set_mask_for_individual_horizon      [1] = no
AHFinderDirect::reset_horizon_after_not_finding      [1] = no
AHFinderDirect::track_origin_from_grid_scalar        [1] = yes
AHFinderDirect::track_origin_source_x                [1] = "PunctureTracker::pt_loc_x[0]"
AHFinderDirect::track_origin_source_y                [1] = "PunctureTracker::pt_loc_y[0]"
AHFinderDirect::track_origin_source_z                [1] = "PunctureTracker::pt_loc_z[0]"
AHFinderDirect::max_allowable_horizon_radius         [1] = 3

AHFinderDirect::origin_x                             [2] = $xm
AHFinderDirect::initial_guess__coord_sphere__x_center[2] = $xm
AHFinderDirect::initial_guess__coord_sphere__radius  [2] = $rm
AHFinderDirect::which_surface_to_store_info          [2] = 3
AHFinderDirect::set_mask_for_individual_horizon      [2] = no
AHFinderDirect::reset_horizon_after_not_finding      [2] = no
AHFinderDirect::track_origin_from_grid_scalar        [2] = yes
AHFinderDirect::track_origin_source_x                [2] = "PunctureTracker::pt_loc_x[1]"
AHFinderDirect::track_origin_source_y                [2] = "PunctureTracker::pt_loc_y[1]"
AHFinderDirect::track_origin_source_z                [2] = "PunctureTracker::pt_loc_z[1]"
AHFinderDirect::max_allowable_horizon_radius         [2] = 3

AHFinderDirect::origin_x                             [3] = 0
AHFinderDirect::find_after_individual                [3] = $find_cah
AHFinderDirect::initial_guess__coord_sphere__x_center[3] = 0
AHFinderDirect::initial_guess__coord_sphere__radius  [3] = 1.0
AHFinderDirect::which_surface_to_store_info          [3] = 4
AHFinderDirect::set_mask_for_individual_horizon      [3] = no
AHFinderDirect::max_allowable_horizon_radius         [3] = 6

################################################################################
# Spherical surfaces
################################################################################

SphericalSurface::nsurfaces             = 5
SphericalSurface::maxntheta             = 66
SphericalSurface::maxnphi               = 124
SphericalSurface::verbose               = no

# Surfaces 0 and 1 are used by PunctureTracker

# Horizon 1
SphericalSurface::ntheta            [2] = 41
SphericalSurface::nphi              [2] = 80
SphericalSurface::nghoststheta      [2] = 2
SphericalSurface::nghostsphi        [2] = 2

# Horizon 2
SphericalSurface::ntheta            [3] = 41
SphericalSurface::nphi              [3] = 80
SphericalSurface::nghoststheta      [3] = 2
SphericalSurface::nghostsphi        [3] = 2

# Common horizon
SphericalSurface::ntheta            [4] = 41
SphericalSurface::nphi              [4] = 80
SphericalSurface::nghoststheta      [4] = 2
SphericalSurface::nghostsphi        [4] = 2

################################################################################
# Isolated Horizons
################################################################################

QuasiLocalMeasures::verbose                = no
QuasiLocalMeasures::veryverbose            = no
QuasiLocalMeasures::interpolator           = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options   = "order=4"
QuasiLocalMeasures::spatial_order          = 4
QuasiLocalMeasures::num_surfaces           = 3
QuasiLocalMeasures::surface_index      [0] = 2
QuasiLocalMeasures::surface_index      [1] = 3
QuasiLocalMeasures::surface_index      [2] = 4
QuasiLocalMeasures::output_vtk_every       = $out3d_every

################################################################################
# Correctness checking
################################################################################

Carpet::poison_new_timelevels           = no
Carpet::check_for_poison                = no

NaNChecker::check_every                 = 256
NanChecker::check_after                 = 0
NaNChecker::report_max                  = 10
NaNChecker::verbose                     = "all"
NaNChecker::action_if_found             = terminate
NaNChecker::out_NaNmask                 = yes
NaNChecker::check_vars                  = "
  ML_BSSN::ML_log_confac
"

################################################################################
# Timers
################################################################################

Carpet::output_timer_tree_every         = 1024
Carpet::output_initialise_timer_tree    = yes

################################################################################
# Output
################################################################################

IO::out_dir                             = "@SIMULATION_NAME@"
IOScalar::one_file_per_group            = yes
IOASCII::one_file_per_group             = yes

IOBasic::outInfo_every                  = 1
IOBasic::outInfo_reductions             = "minimum maximum"
IOBasic::outInfo_vars                   = "
  ML_BSSN::ML_log_confac
  Carpet::physical_time_per_hour
  SystemStatistics::maxrss_mb
  SystemStatistics::swap_used_mb
"

IOScalar::outScalar_every               = 256
IOScalar::outScalar_reductions          = "minimum maximum average"
IOScalar::outScalar_vars                = "SystemStatistics::process_memory_mb"

IOASCII::out0D_every                    = 256
IOASCII::out0D_vars                     = "
  Carpet::timing
  PunctureTracker::pt_loc
  QuasiLocalMeasures::qlm_scalars{out_every = $horizon_every}
"

IOASCII::out1D_every                    = 0
IOASCII::out1D_d                        = no
IOASCII::out1D_x                        = yes
IOASCII::out1D_y                        = no
IOASCII::out1D_z                        = yes
IOASCII::out1D_vars                     = "
  ML_BSSN::ML_log_confac
  ML_BSSN::ML_trace_curv
  WeylScal4::Psi4r
"

IOASCII::out2D_every                    = 0
IOASCII::out2D_vars                     = "
"

IOHDF5::out_every                       = $out3d_every
IOHDF5::out_vars                        = "
  Grid::Coordinates{out_every=1000000000 refinement_levels={0}}
  ML_BSSN::ML_log_confac
  WeylScal4::Psi4r
  WeylScal4::Psi4i
  WeylScal4::curvIr{refinement_levels={3 5}}
  WeylScal4::curvIi{refinement_levels={3 5}}
  WeylScal4::curvJr{refinement_levels={3 5}}
  WeylScal4::curvJi{refinement_levels={3 5}}
"

IOHDF5::out2D_every                     = $out2d_every
IOHDF5::out2D_vars                      = "
  Grid::Coordinates{out_every=1000000000 refinement_levels={0}}
  ML_BSSN::alpha
  ML_BSSN::phi
  ML_BSSN::trK
  WeylScal4::Psi4r
  WeylScal4::Psi4i
"

################################################################################
# Checkpointing and recovery
################################################################################

CarpetIOHDF5::checkpoint                    = yes
IO::checkpoint_ID                           = no
IO::recover                                 = "autoprobe"
IO::out_proc_every                          = 2
IO::checkpoint_on_terminate                 = yes
IO::checkpoint_dir                          = "../checkpoints"
IO::recover_dir                             = "../checkpoints"
IO::abort_on_io_errors                      = yes
CarpetIOHDF5::open_one_input_file_at_a_time = yes
CarpetIOHDF5::compression_level             = 0

################################################################################
# Run termination
################################################################################

TerminationTrigger::max_walltime                 = @WALLTIME_HOURS@
# Trigger termination 30 minutes before the walltime is reached
TerminationTrigger::on_remaining_walltime        = 30
TerminationTrigger::output_remtime_every_minutes = 30
TerminationTrigger::termination_from_file        = yes
TerminationTrigger::termination_file             = "terminate.txt"
TerminationTrigger::create_termination_file      = yes

Cactus::terminate                               = time
Cactus::cctk_final_time                         = $final_time
"""

open(re.sub(r'(.*)\.rpar$', r'\1.par', sys.argv[0]), 'w').write(re.sub(r'\n *',r'\n',Template(Template(lines).substitute(locals())).substitute(locals())))
-------------- next part --------------
Traceback (most recent call last):
  File "/system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar", line 126, in <module>
    sphere_outer_radius = int((outermost_detector + final_time)/(i*hr))*i*hr
ZeroDivisionError: float division by zero
Error: Error while executing parameter file script /system/user/crangano/simulations/GW150914_28/output-0000/GW150914.rpar
Aborting Simfactory.
-------------- next part --------------
Simulation name: GW150914_28
Running simulation GW150914_28
-------------- next part --------------

################################################################################
# Script variables
################################################################################

# D                   = 10.0
# M                   = 1.0
# Pmx                 = 0.00084541526517121
# Pmy                 = -0.09530152296974252
# Pmz                 = 0
# Pphi                = 0.09530152296974252
# Ppx                 = -0.00084541526517121
# Ppy                 = 0.09530152296974252
# Ppz                 = 0
# Pr                  = -0.00084541526517121
# Smx                 = 0.0
# Smy                 = 0.0
# Smz                 = -0.09156449704142013
# Spx                 = 0.0
# Spy                 = 0.0
# Spz                 = 0.09509112426035504
# __file__            = /system/user/crangano/einstein_toolkit/Cactus/repos/einsteinexamples/par/GW150914/GW150914.rpar
# __name__            = __main__
# ahrm                = 0.4461538461538462
# ahrp                = 0.5538461538461539
# center_offset       = -0.5384615384615383
# chim                = -0.46
# chip                = 0.31
# col_width           = 19
# dt0                 = 0.2753406593406593
# dt_it               = 0.002151098901098901
# dtdx                = 0.45
# dtdx0               = 0.225
# e                   = 2.718281828459045
# expected_merger     = 1000.0
# final_time          = 1700.0
# find_cah            = 371902
# h0                  = 1.2237362637362637
# h0_min              = 1.4276923076923076
# half_D              = 5.0
# hfm_min             = 0.022307692307692306
# horizon_every       = 256
# hr                  = 1.2237362637362637
# hr_min              = 1.92
# i                   = 7
# inf                 = inf
# key                 = xp
# levelsm             = [0,17.132308,8.566154,4.283077,2.141538,1.070769,0.535385]
# levelsp             = [0,21.267692,10.633846,5.316923,2.658462,1.329231,0.664615]
# maxrls              = 9
# mm                  = 0.4461538461538462
# mp                  = 0.5538461538461539
# n                   = 28
# n_angular           = 28
# n_min               = 24
# nan                 = nan
# out2d_every         = 1024
# out3d_every         = 0
# out_every           = 128
# outermost_detector  = 500.0
# pi                  = 3.141592653589793
# q                   = 1.2413793103448276
# rl0_every           = 128
# rl1_every           = 128
# rl_max              = 8
# rlsm                = 7
# rlsp                = 7
# rm                  = 0.5353846153846153
# rp                  = 0.6646153846153846
# sphere_inner_radius = 51.39692307692307
# sphere_outer_radius = 2194.1591208791206
# tau                 = 6.283185307179586
# time_after_merger   = 200.0
# val                 = 4.461538461538462
# wave_extract_every  = 256
# waveform_length     = 1200.0
# xm                  = -5.538461538461538
# xp                  = 4.461538461538462

################################################################################
# Active thorns
################################################################################

ActiveThorns = "
ADMBase
ML_ADMConstraints
ADMCoupling
ADMMacros
AEILocalInterp
AHFinderDirect
Boundary
Carpet
CarpetIOASCII
CarpetIOBasic
CarpetIOHDF5
CarpetIOScalar
CarpetInterp
CarpetInterp2
CarpetLib
CarpetReduce
CarpetRegrid2
CarpetTracker
CartGrid3D
CoordBase
CoordGauge
Coordinates
CoordinatesSymmetry
Formaline
GlobalDerivative
hwloc
IOUtil
InitBase
Interpolate2
QuasiLocalMeasures
LocalInterp
LoopControl
MoL
NaNChecker
PunctureTracker
Slab
SpaceMask
SphericalSurface
StaticConformal
SummationByParts
SymBase
SystemStatistics
SystemTopology
TerminationTrigger
TensorTypes
Time
TmunuBase
TwoPunctures
Vectors
ML_BSSN
ML_BSSN_Helper
NewRad
GenericFD
WeylScal4
Multipole
#  WaveExtractCPM
ADMDerivatives
"

################################################################################
# Grid structure
################################################################################

Carpet::domain_from_multipatch          = yes
CartGrid3D::type                        = "multipatch"
CartGrid3D::set_coordinate_ranges_on    = "all maps"
Coordinates::coordinate_system          = "Thornburg04"
Coordinates::h_cartesian                = 1.2237362637362637
Coordinates::h_radial                   = 1.2237362637362637

Coordinates::sphere_inner_radius        = 51.39692307692307
Coordinates::sphere_outer_radius        = 2194.1591208791206
Coordinates::n_angular                  = 28

Driver::ghost_size                      = 5
Coordinates::patch_boundary_size        = 5
Coordinates::additional_overlap_size    = 3
Coordinates::outer_boundary_size        = 5

# change these (to their defaults) to disable z-zymmetry
CoordinatesSymmetry::reflection_z       = yes
CoordinatesSymmetry::stagger            = no
Coordinates::symmetry                   = "+z bitant"
Coordinates::additional_symmetry_size   = 1
Coordinates::verbose                    = no

Time::timestep_method                   = "given"
Time::timestep                          = 0.2753406593406593
Carpet::time_refinement_factors         = "[1,1,2,4,8,16,32,64,128,256]"

################################################################################
# Mesh refinement
################################################################################

Carpet::max_refinement_levels           = 9
CarpetRegrid2::num_centres              = 2
CarpetRegrid2::num_levels_1             = 7
CarpetRegrid2::position_x_1             = 4.461538461538462
CarpetRegrid2::radius_1                 = [0,21.267692,10.633846,5.316923,2.658462,1.329231,0.664615]
CarpetRegrid2::num_levels_2             = 7
CarpetRegrid2::position_x_2             = -5.538461538461538
CarpetRegrid2::radius_2                 = [0,17.132308,8.566154,4.283077,2.141538,1.070769,0.535385]

Carpet::use_buffer_zones                = yes
Carpet::prolongation_order_space        = 5
Carpet::prolongation_order_time         = 2

CarpetRegrid2::regrid_every             = 128
CarpetRegrid2::verbose                  = no
Carpet::grid_coordinates_filename       = "carpet-grid.asc"

################################################################################
# Initial data
################################################################################

ADMBase::initial_data                   = "twopunctures"
ADMBase::initial_lapse                  = "twopunctures-averaged"
ADMBase::initial_shift                  = "zero"
ADMBase::initial_dtlapse                = "zero"
ADMBase::initial_dtshift                = "zero"

TwoPunctures::target_M_plus             = 0.5538461538461539
TwoPunctures::target_M_minus            = 0.4461538461538462

# Initial guesses for iterative solve
TwoPunctures::par_m_plus                = 0.5538461538461539
TwoPunctures::par_m_minus               = 0.4461538461538462

TwoPunctures::par_b                     = 5.0
TwoPunctures::center_offset[0]          = -0.5384615384615383

TwoPunctures::par_P_plus[0]             = -0.00084541526517121
TwoPunctures::par_P_plus[1]             = 0.09530152296974252
TwoPunctures::par_P_plus[2]             = 0

TwoPunctures::par_P_minus[0]            = 0.00084541526517121
TwoPunctures::par_P_minus[1]            = -0.09530152296974252
TwoPunctures::par_P_minus[2]            = 0

TwoPunctures::par_S_plus[0]             = 0.0
TwoPunctures::par_S_plus[1]             = 0.0
TwoPunctures::par_S_plus[2]             = 0.09509112426035504

TwoPunctures::par_S_minus[0]            = 0.0
TwoPunctures::par_S_minus[1]            = 0.0
TwoPunctures::par_S_minus[2]            = -0.09156449704142013

TwoPunctures::grid_setup_method         = "evaluation"
TwoPunctures::give_bare_mass            = no
TwoPunctures::TP_epsilon                = 1e-6
Carpet::init_fill_timelevels            = yes
InitBase::initial_data_setup_method     = "init_single_level"

################################################################################
# Evolution and boundary
################################################################################

GenericFD::jacobian_group = "Coordinates::jacobian"
GenericFD::jacobian_derivative_group = "Coordinates::jacobian2"
GenericFD::jacobian_identity_map = 0

ADMBase::evolution_method         = "ML_BSSN"
ADMBase::lapse_evolution_method   = "ML_BSSN"
ADMBase::shift_evolution_method   = "ML_BSSN"
ADMBase::dtlapse_evolution_method = "ML_BSSN"
ADMBase::dtshift_evolution_method = "ML_BSSN"

ML_BSSN::fdOrder             = 8

# 1+log slicing requires harmonicN = 1 and harmonicF = 2.0
ML_BSSN::harmonicN           = 1
ML_BSSN::harmonicF           = 2.0

ML_BSSN::ShiftGammaCoeff     = 0.75
ML_BSSN::BetaDriver          = 1.0
ML_BSSN::advectLapse         = 1
ML_BSSN::advectShift         = 1

ML_BSSN::MinimumLapse        = 1.0e-8

# conformalaMethod = 1 for W, 0 for phi
ML_BSSN::conformalMethod     = 1

# We apply dissipation using GlobalDerivaitive so disable it here
ML_BSSN::epsDiss             = 0.0

ML_BSSN::dt_lapse_shift_method = "noLapseShiftAdvection"

ML_BSSN::initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::rhs_boundary_condition     = "scalar"
Boundary::radpower                  = 2

################################################################################
# BH tracking
################################################################################

CarpetTracker::surface                      [0] = 0
CarpetTracker::surface                      [1] = 1
PunctureTracker::track                      [0] = yes
PunctureTracker::initial_x                  [0] = 4.461538461538462
PunctureTracker::which_surface_to_store_info[0] = 0
PunctureTracker::track                      [1] = yes
PunctureTracker::initial_x                  [1] = -5.538461538461538
PunctureTracker::which_surface_to_store_info[1] = 1

################################################################################
# Spatial finite differencing
################################################################################

SummationByParts::order                              = 8

# Drop order instead of using upwinded stencils, only for advection derivatives
SummationByParts::sbp_upwind_deriv = no

SummationByParts::sbp_1st_deriv                      = yes
SummationByParts::sbp_2nd_deriv                      = no
SummationByParts::onesided_interpatch_boundaries     = no
SummationByParts::onesided_outer_boundaries          = yes
SummationByParts::use_dissipation                    = no
GlobalDerivative::use_dissipation                    = yes
SummationByParts::scale_with_h                       = yes
SummationByParts::dissipation_type                   = "Kreiss-Oliger"
SummationByParts::epsdis                             = 0.15

# Because dt/dx is half that of the other levels we also need half the dissipation
GlobalDerivative::epsdis_for_level               [0] = 0.075

# Variables for dissipation
SummationByParts::vars                               = "
ML_BSSN::ML_log_confac
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
"

################################################################################
# Time integration
################################################################################

MoL::ODE_Method                         = "rk4"
MoL::MoL_Intermediate_Steps             = 4
MoL::MoL_Num_Scratch_Levels             = 1

################################################################################
# Interpolation
################################################################################

CarpetInterp::check_tree_search         = no
CarpetInterp::tree_search               = yes
# Use 5-th order interpatch interpolation on the Llama grid
Interpolate::interpolator_order         = 5

################################################################################
# Psi4 computation
################################################################################

WeylScal4::fdOrder                   = 8
WeylScal4::calc_scalars              = "psis"
WeylScal4::calc_invariants           = "always"

################################################################################
# Psi4 mode decomposition
################################################################################

# Radii are chosen to be evenly spaced in 1/r as that is the variable
# extrapolation is performed in
Multipole::nradii       = 7
Multipole::radius[0]    = 100
Multipole::radius[1]    = 115
Multipole::radius[2]    = 136
Multipole::radius[3]    = 167
Multipole::radius[4]    = 214
Multipole::radius[5]    = 300
Multipole::radius[6]    = 500
Multipole::ntheta       = 120
Multipole::nphi         = 240
Multipole::variables    = "WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='psi4'}"
Multipole::out_every    = 256
Multipole::l_max        = 8
Multipole::output_hdf5  = yes

# Disable ASCII output to avoid creating a large number of files
Multipole::output_ascii = no

################################################################################
# Gauge invariant perturbations of Schwarzschild (ZM-CPM variables)
################################################################################

# WaveExtractCPM::out_every                  = 256
# WaveExtractCPM::use_carpetinterp2          = no
# WaveExtractCPM::calc_when_necessary        = no
# WaveExtractCPM::verbose                    = 0
# WaveExtractCPM::maximum_detector_number    = 7
# WaveExtractCPM::switch_output_format       = 100
# WaveExtractCPM::rsch2_computation          = "average Schwarzschild metric"
# WaveExtractCPM::l_mode                     = 8
# WaveExtractCPM::m_mode                     = 8
# WaveExtractCPM::detector_radius        [0] = 100
# WaveExtractCPM::detector_radius        [1] = 115
# WaveExtractCPM::detector_radius        [2] = 136
# WaveExtractCPM::detector_radius        [3] = 167
# WaveExtractCPM::detector_radius        [4] = 214
# WaveExtractCPM::detector_radius        [5] = 300
# WaveExtractCPM::detector_radius        [6] = 500
# WaveExtractCPM::maxntheta                  = 120
# WaveExtractCPM::maxnphi                    = 240
# WaveExtractCPM::output_hdf5                = yes
# WaveExtractCPM::output_ascii               = no
# WaveExtractCPM::output_h                   = yes
# WaveExtractCPM::output_Psi                 = yes

################################################################################
# Apparent Horizons
################################################################################

AHFinderDirect::N_horizons                               = 3
AHFinderDirect::find_every                               = 256
AHFinderDirect::output_h_every                           = 0
AHFinderDirect::max_Newton_iterations__initial           = 50
AHFinderDirect::max_Newton_iterations__subsequent        = 50
AHFinderDirect::max_allowable_Theta_growth_iterations    = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name               = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars               = "order=4"
AHFinderDirect::surface_interpolator_name                = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars                = "order=4"
AHFinderDirect::verbose_level                            = "physics details"
AHFinderDirect::move_origins                             = yes

AHFinderDirect::origin_x                             [1] = 4.461538461538462
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = 4.461538461538462
AHFinderDirect::initial_guess__coord_sphere__radius  [1] = 0.6646153846153846
AHFinderDirect::which_surface_to_store_info          [1] = 2
AHFinderDirect::set_mask_for_individual_horizon      [1] = no
AHFinderDirect::reset_horizon_after_not_finding      [1] = no
AHFinderDirect::track_origin_from_grid_scalar        [1] = yes
AHFinderDirect::track_origin_source_x                [1] = "PunctureTracker::pt_loc_x[0]"
AHFinderDirect::track_origin_source_y                [1] = "PunctureTracker::pt_loc_y[0]"
AHFinderDirect::track_origin_source_z                [1] = "PunctureTracker::pt_loc_z[0]"
AHFinderDirect::max_allowable_horizon_radius         [1] = 3

AHFinderDirect::origin_x                             [2] = -5.538461538461538
AHFinderDirect::initial_guess__coord_sphere__x_center[2] = -5.538461538461538
AHFinderDirect::initial_guess__coord_sphere__radius  [2] = 0.5353846153846153
AHFinderDirect::which_surface_to_store_info          [2] = 3
AHFinderDirect::set_mask_for_individual_horizon      [2] = no
AHFinderDirect::reset_horizon_after_not_finding      [2] = no
AHFinderDirect::track_origin_from_grid_scalar        [2] = yes
AHFinderDirect::track_origin_source_x                [2] = "PunctureTracker::pt_loc_x[1]"
AHFinderDirect::track_origin_source_y                [2] = "PunctureTracker::pt_loc_y[1]"
AHFinderDirect::track_origin_source_z                [2] = "PunctureTracker::pt_loc_z[1]"
AHFinderDirect::max_allowable_horizon_radius         [2] = 3

AHFinderDirect::origin_x                             [3] = 0
AHFinderDirect::find_after_individual                [3] = 371902
AHFinderDirect::initial_guess__coord_sphere__x_center[3] = 0
AHFinderDirect::initial_guess__coord_sphere__radius  [3] = 1.0
AHFinderDirect::which_surface_to_store_info          [3] = 4
AHFinderDirect::set_mask_for_individual_horizon      [3] = no
AHFinderDirect::max_allowable_horizon_radius         [3] = 6

################################################################################
# Spherical surfaces
################################################################################

SphericalSurface::nsurfaces             = 5
SphericalSurface::maxntheta             = 66
SphericalSurface::maxnphi               = 124
SphericalSurface::verbose               = no

# Surfaces 0 and 1 are used by PunctureTracker

# Horizon 1
SphericalSurface::ntheta            [2] = 41
SphericalSurface::nphi              [2] = 80
SphericalSurface::nghoststheta      [2] = 2
SphericalSurface::nghostsphi        [2] = 2

# Horizon 2
SphericalSurface::ntheta            [3] = 41
SphericalSurface::nphi              [3] = 80
SphericalSurface::nghoststheta      [3] = 2
SphericalSurface::nghostsphi        [3] = 2

# Common horizon
SphericalSurface::ntheta            [4] = 41
SphericalSurface::nphi              [4] = 80
SphericalSurface::nghoststheta      [4] = 2
SphericalSurface::nghostsphi        [4] = 2

################################################################################
# Isolated Horizons
################################################################################

QuasiLocalMeasures::verbose                = no
QuasiLocalMeasures::veryverbose            = no
QuasiLocalMeasures::interpolator           = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options   = "order=4"
QuasiLocalMeasures::spatial_order          = 4
QuasiLocalMeasures::num_surfaces           = 3
QuasiLocalMeasures::surface_index      [0] = 2
QuasiLocalMeasures::surface_index      [1] = 3
QuasiLocalMeasures::surface_index      [2] = 4
QuasiLocalMeasures::output_vtk_every       = 0

################################################################################
# Correctness checking
################################################################################

Carpet::poison_new_timelevels           = no
Carpet::check_for_poison                = no

NaNChecker::check_every                 = 256
NanChecker::check_after                 = 0
NaNChecker::report_max                  = 10
NaNChecker::verbose                     = "all"
NaNChecker::action_if_found             = terminate
NaNChecker::out_NaNmask                 = yes
NaNChecker::check_vars                  = "
ML_BSSN::ML_log_confac
"

################################################################################
# Timers
################################################################################

Carpet::output_timer_tree_every         = 1024
Carpet::output_initialise_timer_tree    = yes

################################################################################
# Output
################################################################################

IO::out_dir                             = "@SIMULATION_NAME@"
IOScalar::one_file_per_group            = yes
IOASCII::one_file_per_group             = yes

IOBasic::outInfo_every                  = 1
IOBasic::outInfo_reductions             = "minimum maximum"
IOBasic::outInfo_vars                   = "
ML_BSSN::ML_log_confac
Carpet::physical_time_per_hour
SystemStatistics::maxrss_mb
SystemStatistics::swap_used_mb
"

IOScalar::outScalar_every               = 256
IOScalar::outScalar_reductions          = "minimum maximum average"
IOScalar::outScalar_vars                = "SystemStatistics::process_memory_mb"

IOASCII::out0D_every                    = 256
IOASCII::out0D_vars                     = "
Carpet::timing
PunctureTracker::pt_loc
QuasiLocalMeasures::qlm_scalars{out_every = 256}
"

IOASCII::out1D_every                    = 0
IOASCII::out1D_d                        = no
IOASCII::out1D_x                        = yes
IOASCII::out1D_y                        = no
IOASCII::out1D_z                        = yes
IOASCII::out1D_vars                     = "
ML_BSSN::ML_log_confac
ML_BSSN::ML_trace_curv
WeylScal4::Psi4r
"

IOASCII::out2D_every                    = 0
IOASCII::out2D_vars                     = "
"

IOHDF5::out_every                       = 0
IOHDF5::out_vars                        = "
Grid::Coordinates{out_every=1000000000 refinement_levels={0}}
ML_BSSN::ML_log_confac
WeylScal4::Psi4r
WeylScal4::Psi4i
WeylScal4::curvIr{refinement_levels={3 5}}
WeylScal4::curvIi{refinement_levels={3 5}}
WeylScal4::curvJr{refinement_levels={3 5}}
WeylScal4::curvJi{refinement_levels={3 5}}
"

IOHDF5::out2D_every                     = 1024
IOHDF5::out2D_vars                      = "
Grid::Coordinates{out_every=1000000000 refinement_levels={0}}
ML_BSSN::alpha
ML_BSSN::phi
ML_BSSN::trK
WeylScal4::Psi4r
WeylScal4::Psi4i
"

################################################################################
# Checkpointing and recovery
################################################################################

CarpetIOHDF5::checkpoint                    = yes
IO::checkpoint_ID                           = no
IO::recover                                 = "autoprobe"
IO::out_proc_every                          = 2
IO::checkpoint_on_terminate                 = yes
IO::checkpoint_dir                          = "../checkpoints"
IO::recover_dir                             = "../checkpoints"
IO::abort_on_io_errors                      = yes
CarpetIOHDF5::open_one_input_file_at_a_time = yes
CarpetIOHDF5::compression_level             = 0

################################################################################
# Run termination
################################################################################

TerminationTrigger::max_walltime                 = @WALLTIME_HOURS@
# Trigger termination 30 minutes before the walltime is reached
TerminationTrigger::on_remaining_walltime        = 30
TerminationTrigger::output_remtime_every_minutes = 30
TerminationTrigger::termination_from_file        = yes
TerminationTrigger::termination_file             = "terminate.txt"
TerminationTrigger::create_termination_file      = yes

Cactus::terminate                               = time
Cactus::cctk_final_time                         = 1700.0


More information about the Users mailing list