[ET Trac] #2861: Running qc0 with CarpetX

Lucas Timotheo Sanches trac-noreply at einsteintoolkit.org
Thu Mar 13 10:51:27 CDT 2025


#2861: Running qc0 with CarpetX

 Reporter: Alejandra Gonzalez
   Status: open
Milestone: 
  Version: 
     Type: bug
 Priority: major
Component: EinsteinToolkit thorn

Comment (by Lucas Timotheo Sanches):

Hi Alejandra.  
  
I was unable to make `Punctures` work by simply modifying parameters, which suggests that the thorn has a more fundamental issue. This will need to be further investigated at another opportune moment.  
  
Regardless, as you noted `TwoPuncturesX` is the preferred option for initial data and should be used instead.  
  
Yes, your output seems to indicate that your execution of the parameter file was successful. As you may have noticed, this example parameter file only runs for a single iteration, which is probably not very useful for your needs.  
  
As for output formats, here’s an abbreviated guideline  
  
- Whenever you intend to visualize 3D data, use the `Silo` format. The `Silo` format can be used natively with [LLNL’s VisIt](https://visit-dav.github.io/visit-website/index.html). This is probably the easiest way to visualize `CarpetX` data, as it allows you to simply read the files and create plots by using a GUI. The `OpenPMD` file is also an option for visualization, but you may have to write custom Python scripts for it.  
- Whenever you wish to store data, in Checkpoints, for instance, the `ADIOS2` and `OpenPMD` files are adequate.  
- `TSV` data, which is plain text, not recommended for storage, in general. It is more of a debug format, where you can look at grid function values in a human readable manner. `TSV` data is also adequate for storing the multipolar extractions of the Weyl scalar and the location of the punctures throughout the simulation.  
  
I have compiled a more complex `qc0` parameter file, which uses `TwoPunctureX` for initial data, but also performs Apparent Horizon finding, puncture tracking and multipole extractions of the Psi4 Weyl scalar. This example should give you a good idea on what production run looks like and may be useful to your applications.  

```
# Settings
#################################
$itlast             = 1000000000
$final_time         = 200
$max_runtime        = 23 * 60 # In minutes

$xmin               = -120.0
$xmax               = +120.0

$ncells             = 120
$nlevels            = 7

$time_stepper       = "RK4"
$dtfac              = 0.25
$error_threshold    = 0.9

$out_every          = 32
$out_silo_every     = 64 * 8
$wave_extract_every = 32
#################################

ActiveThorns = "
  ADMBaseX
  AEILocalInterp
  AHFinderDirect
  BoxInBox
  CarpetX
  CoordinatesX
  IOUtil
  Multipole
  NewRadX
  ODESolvers
  PunctureTracker
  SphericalSurface
  TimerReport
  TmunuBaseX
  TwoPuncturesX
  WeylScal4
  Z4c
"

# Driver
Cactus::cctk_show_schedule       = yes
Cactus::presync_mode             = "mixed-error"
CarpetX::verbose                 = no
CarpetX::poison_undefined_values = no

# Run Termination
Cactus::terminate       = "any"
Cactus::cctk_itlast     = $itlast
Cactus::cctk_final_time = $final_time
Cactus::max_runtime     = $max_runtime

# Grid setup
CarpetX::xmin     = $xmin
CarpetX::ymin     = $xmin
CarpetX::zmin     = $xmin
CarpetX::xmax     = $xmax
CarpetX::ymax     = $xmax
CarpetX::zmax     = $xmax
CarpetX::ncells_x = $ncells
CarpetX::ncells_y = $ncells
CarpetX::ncells_z = $ncells
CarpetX::ghost_size          = 3
CarpetX::interpolation_order = 3
CarpetX::prolongation_type   = "ddf"
CarpetX::prolongation_order  = 5

# Boundary conditions
CarpetX::boundary_x       = "linear extrapolation"
CarpetX::boundary_y       = "linear extrapolation"
CarpetX::boundary_z       = "linear extrapolation"
CarpetX::boundary_upper_x = "linear extrapolation"
CarpetX::boundary_upper_y = "linear extrapolation"
CarpetX::boundary_upper_z = "linear extrapolation"

# Mesh refinement
CarpetX::max_num_levels         = $nlevels
CarpetX::regrid_every           = 128
CarpetX::regrid_error_threshold = $error_threshold
BoxInBox::num_regions  = 2

# Region 1
BoxInBox::shape_1      = "cube"
BoxInBox::num_levels_1 = $nlevels
BoxInBox::position_x_1 = +1.168642873
BoxInBox::radius_1     = [-1., 64., 16., 8., 4., 2., 1.]

# Region 2
BoxInBox::shape_2      = "cube"
BoxInBox::num_levels_2 = $nlevels
BoxInBox::position_x_2 = -1.168642873
BoxInBox::radius_2     = [-1., 64., 16., 8., 4., 2., 1.]

# Initial data
ADMBaseX::initial_data           = "TwoPunctures"
ADMBaseX::initial_lapse          = "TwoPunctures-averaged"
TwoPuncturesX::verbose           = yes
TwoPuncturesX::par_b             = 1.168642873
TwoPuncturesX::par_m_plus        = 0.453
TwoPuncturesX::par_m_minus       = 0.453
TwoPuncturesX::par_P_plus [1]    = +0.3331917498
TwoPuncturesX::par_P_minus[1]    = -0.3331917498
TwoPuncturesX::grid_setup_method = "evaluation"
TwoPuncturesX::TP_epsilon        = 1.0e-2
TwoPuncturesX::TP_Tiny           = 1.0e-2

# Time Integration
CarpetX::dtfac      = $dtfac
ODESolvers::verbose = no
ODESolvers::method  = $time_stepper

# Spacetime evolution
Z4c::calc_ADM_vars       = yes #Required for computing the Weyl scalar
Z4c::calc_ADMRHS_vars    = yes #Required for computing the Weyl scalar
Z4c::calc_constraints    = yes
Z4c::chi_floor           = 1.0e-6
Z4c::alphaG_floor        = 1.0e-8
Z4c::epsdiss             = 0.32
Z4C::boundary_conditions = "NewRadX"

# Spherical surfaces
SphericalSurface::nsurfaces = 5
SphericalSurface::maxntheta = 39
SphericalSurface::maxnphi   = 76
SphericalSurface::verbose   = no

# Surfaces 0 and 1 are used by PunctureTracker

# Tracker 1
SphericalSurface::ntheta           [0] = 39
SphericalSurface::nphi             [0] = 76
SphericalSurface::nghoststheta     [0] = 2
SphericalSurface::nghostsphi       [0] = 2

# Tracker 2
SphericalSurface::ntheta           [1] = 39
SphericalSurface::nphi             [1] = 76
SphericalSurface::nghoststheta     [1] = 2
SphericalSurface::nghostsphi       [1] = 2

# Surfaces 2, 3, 4 are used by AHFinderDirect and QuasiLocalMeasures
# Horizon 1
 SphericalSurface::ntheta           [2] = 39
 SphericalSurface::nphi             [2] = 76
 SphericalSurface::nghoststheta     [2] = 2
 SphericalSurface::nghostsphi       [2] = 2

# Horizon 2
 SphericalSurface::ntheta           [3] = 39
 SphericalSurface::nphi             [3] = 76
 SphericalSurface::nghoststheta     [3] = 2
 SphericalSurface::nghostsphi       [3] = 2
 
# Horizon 3
 SphericalSurface::ntheta           [4] = 39
 SphericalSurface::nphi             [4] = 76
 SphericalSurface::nghoststheta     [4] = 2
 SphericalSurface::nghostsphi       [4] = 2

# Puncture tracking 
PunctureTracker::verbose                        = no
PunctureTracker::track_boxes                    = yes
PunctureTracker::interp_order                   = 3
PunctureTracker::track                      [0] = yes
PunctureTracker::initial_x                  [0] = +1.168642873
PunctureTracker::track                      [1] = yes
PunctureTracker::initial_x                  [1] = -1.168642873

# Apparent horizons
 AHFinderDirect::find_every = $wave_extract_every 
 AHFinderDirect::verbose_level                            = "physics details"
 AHFinderDirect::output_BH_diagnostics                    = "true"
 AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES       = no 
 AHFinderDirect::output_h_every = 0  # Turn off horizon shape output
 
 AHFinderDirect::max_Newton_iterations__initial           = 50
 AHFinderDirect::max_Newton_iterations__subsequent        = 50
 AHFinderDirect::max_allowable_Theta_growth_iterations    = 10
 AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
 AHFinderDirect::geometry_interpolator_name               = "CarpetX"
 AHFinderDirect::geometry_interpolator_pars = "order=3 boundary_off_centering_tolerance={1.0e-10 1.0e-10 1.0e-10 1.0e-10 1.0e-10 1.0e-10} boundary_extrapolation_tolerance={0.0 0.0 0.0 0.0 0.0 0.0}"
 AHFinderDirect::surface_interpolator_name  = "Lagrange polynomial interpolation"
 AHFinderDirect::surface_interpolator_pars  = "order=3"
 
 AHFinderDirect::move_origins            = yes 
 AHFinderDirect::reshape_while_moving    = yes 
 AHFinderDirect::predict_origin_movement = yes 
 
 AHFinderDirect::N_horizons = 3
 
 AHFinderDirect::origin_x                             [1] = +1.168642873
 AHFinderDirect::origin_y                             [1] = 0
 AHFinderDirect::origin_z                             [1] = 0
 AHFinderDirect::initial_guess_method                 [1] = "coordinate sphere"
 AHFinderDirect::initial_guess__coord_sphere__x_center[1] = +1.168642873
 AHFinderDirect::initial_guess__coord_sphere__y_center[1] = 0
 AHFinderDirect::initial_guess__coord_sphere__z_center[1] = 0
 AHFinderDirect::initial_guess__coord_sphere__radius  [1] = 0.25
 AHFinderDirect::which_surface_to_store_info          [1] = 2 
 AHFinderDirect::set_mask_for_individual_horizon      [1] = no
 AHFinderDirect::reset_horizon_after_not_finding      [1] = no
 AHFinderDirect::dont_find_after_individual_time      [1] = 30.0 
 
 AHFinderDirect::origin_x                             [2] = -1.168642873
 AHFinderDirect::origin_y                             [2] = 0
 AHFinderDirect::origin_z                             [2] = 0
 AHFinderDirect::initial_guess_method                 [2] = "coordinate sphere"
 AHFinderDirect::initial_guess__coord_sphere__x_center[2] = -1.168642873
 AHFinderDirect::initial_guess__coord_sphere__y_center[2] = 0
 AHFinderDirect::initial_guess__coord_sphere__z_center[2] = 0
 AHFinderDirect::initial_guess__coord_sphere__radius  [2] =  0.25
 AHFinderDirect::which_surface_to_store_info          [2] = 3 
 AHFinderDirect::reset_horizon_after_not_finding      [2] = no
 AHFinderDirect::dont_find_after_individual_time      [2] = 30.0
 
 AHFinderDirect::origin_x                             [3] = 0.0
 AHFinderDirect::origin_y                             [3] = 0
 AHFinderDirect::origin_z                             [3] = 0
 AHFinderDirect::initial_guess_method                 [3] = "coordinate sphere"
 AHFinderDirect::initial_guess__coord_sphere__x_center[3] = 0.0
 AHFinderDirect::initial_guess__coord_sphere__y_center[3] = 0
 AHFinderDirect::initial_guess__coord_sphere__z_center[3] = 0
 AHFinderDirect::initial_guess__coord_sphere__radius  [3] = 0.6 
 AHFinderDirect::which_surface_to_store_info          [3] = 4 
 AHFinderDirect::find_after_individual_time           [3] = 0.0

# Wave extraction
WeylScal4::WeylScal4_psi4_calc_4th_calc_every = $wave_extract_every 

Multipole::verbose           = no
Multipole::nradii            = 8
Multipole::radius[0]         = 15
Multipole::radius[1]         = 30
Multipole::radius[2]         = 40
Multipole::radius[3]         = 50
Multipole::radius[4]         = 60
Multipole::radius[5]         = 70
Multipole::radius[6]         = 80
Multipole::radius[7]         = 90
Multipole::ntheta            = 120
Multipole::nphi              = 240
Multipole::variables         = "
  WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='Psi4'}
"
Multipole::out_every         = $wave_extract_every
Multipole::l_max             = 4
Multipole::interpolator_pars = "order=3"

# Checkpoint and recovery
IO::out_dir                         = $parfile
IO::out_every                       = $out_every
IO::parfile_write                   = no
IO::checkpoint_dir                  = "checkpoints_$parfile"
IO::recover_dir                     = "checkpoints_$parfile"
IO::checkpoint_ID                   = no
IO::checkpoint_every_walltime_hours = 6
IO::checkpoint_on_terminate         = yes
IO::recover                         = "autoprobe"
CarpetX::checkpoint_method          = "openpmd"
CarpetX::recover_method             = "openpmd"

# Output
TimerReport::out_every              = $out_every
TimerReport::out_filename           = "TimerReport"
TimerReport::output_schedule_timers = no
TimerReport::n_top_timers           = 100

# Norm and reduction output
CarpetX::out_norm_vars          = "all"
CarpetX::out_norm_omit_unstable = yes

# 1D output
CarpetX::out_tsv_vars = "
  PunctureTracker::pt_loc
  PunctureTracker::pt_vel
"

# 3D output
CarpetX::out_silo_every = $out_silo_every
CarpetX::out_silo_vars = "
  ADMBaseX::alp
"
```

‌

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2861/running-qc0-with-carpetx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/trac/attachments/20250313/7f57eac4/attachment-0001.htm>


More information about the Trac mailing list