[Users] McLachlan help
James Healy
jchsma at rit.edu
Mon Jul 27 16:43:24 CDT 2015
Hello,
I have been running some tests on Stampede comparing the run speed of
McLachlan to RIT's evolution thorn LazEv. I started with the
qc0-mclachlan.par parameter file included with the Einstein Toolkit,
added a few refinement levels, increased the resolution and changed
McLachlan to be 8th order (and increased the number of ghost zones to
5). I also increased the initial separation so the finest grids aren't
already overlapping. To compare with LazEv, I removed the McLachlan and
Dissipation thorns and replaced them with LazEv. Everything else in the
parameter file is exactly the same. I tried using both the McLachlan
master and rewrite branches.
The grid setup is 10 levels of refinement, dx=4M on the coarsest with
outer boundary at 400M, M/128 on the finest with r=0.6M, CFL is 0.25.
Both use 8th order spatial differencing with ghost_size=5 and 5th order
dissipation.
Below is a summary of the results as reported at iteration 256 from
Carpet::physical_time_per_hour:
McLachlan - rewrite branch: 3.0596110 M/hr
McLachlan - master branch: 3.8033607 M/hr
LazEv - 4.1941544 M/hr
I am using the stampede-impi.cfg configuration file in simfactory.
"module list" returns:
1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7) TACC
2) Linux 4) intel/13.0.2.146 6) cluster 8) impi/4.1.0.030
Attached is my parameter file. I pasted the McLachlan parameters
below. Are there any optimizations that I can use for McLachlan? Are
the parameters I am using for it what would be used for production runs?
ML_BSSN::harmonicN = 1 # 1+log
ML_BSSN::harmonicF = 2.0 # 1+log
ML_BSSN::ShiftGammaCoeff = 0.75
ML_BSSN::BetaDriver = 1.0
ML_BSSN::LapseAdvectionCoeff = 1.0
ML_BSSN::ShiftAdvectionCoeff = 1.0
ML_BSSN::MinimumLapse = 1.0e-8
ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::my_rhs_boundary_condition = "NewRad"
Boundary::radpower = 2
ML_BSSN::ML_log_confac_bound = "none"
ML_BSSN::ML_metric_bound = "none"
ML_BSSN::ML_Gamma_bound = "none"
ML_BSSN::ML_trace_curv_bound = "none"
ML_BSSN::ML_curv_bound = "none"
ML_BSSN::ML_lapse_bound = "none"
ML_BSSN::ML_dtlapse_bound = "none"
ML_BSSN::ML_shift_bound = "none"
ML_BSSN::ML_dtshift_bound = "none"
ML_BSSN::fdOrder = 8
ActiveThorns = "Dissipation"
Dissipation::order = 5
Dissipation::vars = "
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
"
ActiveThorns = "ML_ADMConstraints"
Thanks,
Jim Healy
-------------- next part --------------
Cactus::cctk_run_title = "QC-0"
Cactus::cctk_full_warnings = yes
Cactus::highlight_warning_messages = no
Cactus::terminate = "time"
Cactus::cctk_final_time = 200.0
ActiveThorns = "IOUtil"
IO::out_dir = $parfile
ActiveThorns = "AEILocalInterp"
#ActiveThorns = "BLAS LAPACK"
ActiveThorns = "Fortran"
ActiveThorns = "GSL"
ActiveThorns = "GenericFD"
ActiveThorns = "HDF5"
ActiveThorns = "LocalInterp"
ActiveThorns = "LoopControl"
ActiveThorns = "Slab"
ActiveThorns = "TGRtensor"
ActiveThorns = "SummationByParts"
SummationByParts::order = 4
ActiveThorns = "InitBase"
ActiveThorns = "Carpet CarpetLib CarpetInterp CarpetReduce CarpetSlab"
Carpet::verbose = no
Carpet::veryverbose = no
Carpet::schedule_barriers = no
Carpet::storage_verbose = no
#Carpet::timers_verbose = no
CarpetLib::output_bboxes = no
Carpet::domain_from_coordbase = yes
Carpet::max_refinement_levels = 10
driver::ghost_size = 5
Carpet::use_buffer_zones = yes
Carpet::prolongation_order_space = 5
Carpet::prolongation_order_time = 2
Carpet::convergence_level = 0
Carpet::init_fill_timelevels = no
Carpet::init_3_timelevels = yes
Carpet::poison_new_timelevels = yes
CarpetLib::poison_new_memory = yes
Carpet::output_timers_every = 5120
CarpetLib::print_timestats_every = 5120
CarpetLib::print_memstats_every = 5120
ActiveThorns = "NaNChecker"
NaNChecker::check_every = 1 # 512
#NaNChecker::verbose = "all"
#NaNChecker::action_if_found = "just warn"
NaNChecker::action_if_found = "terminate"
NaNChecker::check_vars = "
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
ADMBase::metric
ADMBase::curv
ADMBase::lapse
ADMBase::shift
ADMBase::dtlapse
ADMBase::dtshift
"
ActiveThorns = "Boundary CartGrid3D CoordBase ReflectionSymmetry RotatingSymmetry180 SymBase"
CoordBase::domainsize = "minmax"
CoordBase::xmin = 0.00
CoordBase::ymin = -400.00
CoordBase::zmin = 0.00
CoordBase::xmax = +400.00
CoordBase::ymax = +400.00
CoordBase::zmax = +400.00
CoordBase::dx = 4.00
CoordBase::dy = 4.00
CoordBase::dz = 4.00
CoordBase::boundary_size_x_lower = 5
CoordBase::boundary_size_y_lower = 5
CoordBase::boundary_size_z_lower = 5
CoordBase::boundary_size_x_upper = 5
CoordBase::boundary_size_y_upper = 5
CoordBase::boundary_size_z_upper = 5
CoordBase::boundary_shiftout_x_lower = 1
CoordBase::boundary_shiftout_z_lower = 1
CartGrid3D::type = "coordbase"
ReflectionSymmetry::reflection_z = yes
ReflectionSymmetry::avoid_origin_z = no
ActiveThorns = "SphericalSurface"
SphericalSurface::nsurfaces = 6
SphericalSurface::maxntheta = 39
SphericalSurface::maxnphi = 76
SphericalSurface::ntheta [0] = 39
SphericalSurface::nphi [0] = 76
SphericalSurface::nghoststheta[0] = 2
SphericalSurface::nghostsphi [0] = 2
SphericalSurface::ntheta [1] = 39
SphericalSurface::nphi [1] = 76
SphericalSurface::nghoststheta[1] = 2
SphericalSurface::nghostsphi [1] = 2
SphericalSurface::ntheta [2] = 39
SphericalSurface::nphi [2] = 76
SphericalSurface::nghoststheta[2] = 2
SphericalSurface::nghostsphi [2] = 2
SphericalSurface::ntheta [3] = 39
SphericalSurface::nphi [3] = 76
SphericalSurface::nghoststheta[3] = 2
SphericalSurface::nghostsphi [3] = 2
SphericalSurface::ntheta [4] = 39
SphericalSurface::nphi [4] = 76
SphericalSurface::nghoststheta[4] = 2
SphericalSurface::nghostsphi [4] = 2
SphericalSurface::ntheta [5] = 39
SphericalSurface::nphi [5] = 76
SphericalSurface::nghoststheta[5] = 2
SphericalSurface::nghostsphi [5] = 2
ActiveThorns = "CarpetMask"
CarpetMask::verbose = no
CarpetMask::excluded_surface [0] = 0
CarpetMask::excluded_surface_factor[0] = 1.0
CarpetMask::excluded_surface [1] = 1
CarpetMask::excluded_surface_factor[1] = 1.0
CarpetMask::excluded_surface [2] = 2
CarpetMask::excluded_surface_factor[2] = 1.0
ActiveThorns = "CarpetRegrid2 CarpetTracker"
CarpetTracker::surface[0] = 0
CarpetTracker::surface[1] = 1
CarpetTracker::surface[2] = 2
CarpetRegrid2::regrid_every = 128
CarpetRegrid2::freeze_unaligned_levels = yes
CarpetRegrid2::symmetry_rotating180 = yes
CarpetRegrid2::verbose = yes
CarpetRegrid2::num_centres = 3
CarpetRegrid2::num_levels_1 = 10
CarpetRegrid2::position_x_1 = +5.168642873
CarpetRegrid2::radius_1[ 1] = 200.0
CarpetRegrid2::radius_1[ 2] = 100.0
CarpetRegrid2::radius_1[ 3] = 50.0
CarpetRegrid2::radius_1[ 4] = 16.0
CarpetRegrid2::radius_1[ 5] = 8.0
CarpetRegrid2::radius_1[ 6] = 4.0
CarpetRegrid2::radius_1[ 7] = 2.0
CarpetRegrid2::radius_1[ 8] = 1.0
CarpetRegrid2::radius_1[ 9] = 0.6
CarpetRegrid2::movement_threshold_1 = 0.16
CarpetRegrid2::num_levels_2 = 10
CarpetRegrid2::position_x_2 = -5.168642873
CarpetRegrid2::radius_2[ 1] = 200.0
CarpetRegrid2::radius_2[ 2] = 100.0
CarpetRegrid2::radius_2[ 3] = 50.0
CarpetRegrid2::radius_2[ 4] = 16.0
CarpetRegrid2::radius_2[ 5] = 8.0
CarpetRegrid2::radius_2[ 6] = 4.0
CarpetRegrid2::radius_2[ 7] = 2.0
CarpetRegrid2::radius_2[ 8] = 1.0
CarpetRegrid2::radius_2[ 9] = 0.6
CarpetRegrid2::movement_threshold_2 = 0.16
CarpetRegrid2::num_levels_3 = 10
CarpetRegrid2::active_3 = no
CarpetRegrid2::radius_3[ 1] = 200.0
CarpetRegrid2::radius_3[ 2] = 100.0
CarpetRegrid2::radius_3[ 3] = 50.0
CarpetRegrid2::radius_3[ 4] = 16.0
CarpetRegrid2::radius_3[ 5] = 8.0
CarpetRegrid2::radius_3[ 6] = 4.0
CarpetRegrid2::radius_3[ 7] = 2.0
CarpetRegrid2::radius_3[ 8] = 1.0
CarpetRegrid2::radius_3[ 9] = 0.6
CarpetRegrid2::movement_threshold_3 = 0.16
ActiveThorns = "MoL Time"
MoL::ODE_Method = "RK4"
MoL::MoL_Intermediate_Steps = 4
MoL::MoL_Num_Scratch_Levels = 1
#Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32]" #, 64, 128, 256]"
Time::dtfac = 0.25
ActiveThorns = "ADMBase ADMCoupling ADMMacros CoordGauge SpaceMask StaticConformal TmunuBase"
ADMMacros::spatial_order = 4
ActiveThorns = "TwoPunctures"
ADMBase::metric_type = "physical"
initbase::initial_data_setup_method = init_all_levels
ADMBase::initial_data = "twopunctures"
ADMBase::initial_lapse = "twopunctures-averaged"
ADMBase::initial_shift = "zero"
ADMBase::initial_dtlapse = "zero"
ADMBase::initial_dtshift = "zero"
# needed for AHFinderDirect
ADMBase::metric_timelevels = 3
TwoPunctures::par_b = 5.168642873
TwoPunctures::par_m_plus = 0.453
TwoPunctures::par_m_minus = 0.453
TwoPunctures::par_P_plus [1] = +0.1131917498
TwoPunctures::par_P_minus[1] = -0.1131917498
#TODO# TwoPunctures::grid_setup_method = "evaluation"
TwoPunctures::TP_epsilon = 1.0e-2
TwoPunctures::TP_Tiny = 1.0e-2
TwoPunctures::verbose = yes
ActiveThorns = "ML_BSSN ML_BSSN_Helper NewRad"
ADMBase::evolution_method = "ML_BSSN"
ADMBase::lapse_evolution_method = "ML_BSSN"
ADMBase::shift_evolution_method = "ML_BSSN"
ADMBase::dtlapse_evolution_method = "ML_BSSN"
ADMBase::dtshift_evolution_method = "ML_BSSN"
ML_BSSN::harmonicN = 1 # 1+log
ML_BSSN::harmonicF = 2.0 # 1+log
ML_BSSN::ShiftGammaCoeff = 0.75
ML_BSSN::BetaDriver = 1.0
ML_BSSN::LapseAdvectionCoeff = 1.0
ML_BSSN::ShiftAdvectionCoeff = 1.0
ML_BSSN::MinimumLapse = 1.0e-8
ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::my_rhs_boundary_condition = "NewRad"
Boundary::radpower = 2
ML_BSSN::ML_log_confac_bound = "none"
ML_BSSN::ML_metric_bound = "none"
ML_BSSN::ML_Gamma_bound = "none"
ML_BSSN::ML_trace_curv_bound = "none"
ML_BSSN::ML_curv_bound = "none"
ML_BSSN::ML_lapse_bound = "none"
ML_BSSN::ML_dtlapse_bound = "none"
ML_BSSN::ML_shift_bound = "none"
ML_BSSN::ML_dtshift_bound = "none"
ML_BSSN::fdOrder = 8
ActiveThorns = "Dissipation"
Dissipation::order = 5
Dissipation::vars = "
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
"
ActiveThorns = "ML_ADMConstraints"
ActiveThorns = "WeylScal4 Multipole"
WeylScal4::offset = 1e-8
WeylScal4::fd_order = "4th"
WeylScal4::verbose = 0
Multipole::nradii = 8
Multipole::out_every = 128
Multipole::radius[0] = 15
Multipole::radius[1] = 30
Multipole::radius[2] = 40
Multipole::radius[3] = 50
Multipole::radius[4] = 60
Multipole::radius[5] = 70
Multipole::radius[6] = 80
Multipole::radius[7] = 90
Multipole::variables = "WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='Psi4'}"
Multipole::l_max = 4
ActiveThorns = "AHFinderDirect"
AHFinderDirect::find_every = 128
AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES = no
AHFinderDirect::move_origins = yes
AHFinderDirect::reshape_while_moving = yes
AHFinderDirect::predict_origin_movement = yes
AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars = "order=4"
AHFinderDirect::surface_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars = "order=4"
AHFinderDirect::output_h_every = 0
AHFinderDirect::N_horizons = 6
AHFinderDirect::origin_x [1] = +5.168642873
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = +5.168642873
AHFinderDirect::initial_guess__coord_sphere__radius [1] = 0.25
AHFinderDirect::which_surface_to_store_info [1] = 0
AHFinderDirect::reset_horizon_after_not_finding [1] = no
AHFinderDirect::dont_find_after_individual_time [1] = 30.0
AHFinderDirect::origin_x [2] = -5.168642873
AHFinderDirect::initial_guess__coord_sphere__x_center[2] = -5.168642873
AHFinderDirect::initial_guess__coord_sphere__radius [2] = 0.25
AHFinderDirect::which_surface_to_store_info [2] = 1
AHFinderDirect::reset_horizon_after_not_finding [2] = no
AHFinderDirect::dont_find_after_individual_time [2] = 30.0
AHFinderDirect::initial_guess__coord_sphere__radius [3] = 1.0
AHFinderDirect::which_surface_to_store_info [3] = 2
AHFinderDirect::reset_horizon_after_not_finding [3] = no
AHFinderDirect::find_after_individual_time [3] = 15.0
AHFinderDirect::surface_definition [4] = "expansion product"
AHFinderDirect::surface_selection [4] = "areal radius"
AHFinderDirect::desired_value [4] = 50.0
AHFinderDirect::initial_guess__coord_sphere__radius [4] = 50.0
AHFinderDirect::which_surface_to_store_info [4] = 3
AHFinderDirect::reset_horizon_after_not_finding [4] = no
AHFinderDirect::depends_on [5] = 1
AHFinderDirect::desired_value_offset [5] = 0.001
AHFinderDirect::which_surface_to_store_info [5] = 4
AHFinderDirect::reset_horizon_after_not_finding [5] = no
AHFinderDirect::dont_find_after_individual_time [5] = 30.0
AHFinderDirect::depends_on [6] = 3
AHFinderDirect::desired_value_offset [6] = 0.001
AHFinderDirect::which_surface_to_store_info [6] = 5
AHFinderDirect::reset_horizon_after_not_finding [6] = no
AHFinderDirect::find_after_individual_time [6] = 15.0
ActiveThorns = "QuasiLocalMeasures"
QuasiLocalMeasures::verbose = yes
QuasiLocalMeasures::interpolator = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options = "order=4"
QuasiLocalMeasures::spatial_order = 4
QuasiLocalMeasures::num_surfaces = 6
QuasiLocalMeasures::surface_index[0] = 0
QuasiLocalMeasures::surface_index[1] = 1
QuasiLocalMeasures::surface_index[2] = 2
QuasiLocalMeasures::surface_index[3] = 3
QuasiLocalMeasures::surface_index[4] = 4
QuasiLocalMeasures::surface_index[5] = 5
ActiveThorns = "CarpetIOBasic"
IOBasic::outInfo_every = 1
IOBasic::outInfo_reductions = "norm2"
IOBasic::outInfo_vars = "
Carpet::physical_time_per_hour
ML_ADMConstraints::H
SphericalSurface::sf_radius
QuasiLocalMeasures::qlm_spin[0]
"
ActiveThorns = "CarpetIOScalar"
IOScalar::one_file_per_group = yes
IOScalar::outScalar_every = 128
IOScalar::outScalar_vars = "
CarpetReduce::weight
ADMBase::metric
ADMBase::curv
ADMBase::lapse
ADMBase::shift
ADMBase::dtlapse
ADMBase::dtshift
WEYLSCAL4::Psi4r
WEYLSCAL4::Psi4i
ML_ADMConstraints::ML_Ham
ML_ADMConstraints::ML_mom
SphericalSurface::sf_radius
QuasiLocalMeasures::qlm_newman_penrose
QuasiLocalMeasures::qlm_weyl_scalars
QuasiLocalMeasures::qlm_ricci_scalars
QuasiLocalMeasures::qlm_twometric
QuasiLocalMeasures::qlm_killing_vector
QuasiLocalMeasures::qlm_killed_twometric
QuasiLocalMeasures::qlm_invariant_coordinates
QuasiLocalMeasures::qlm_3determinant
"
ActiveThorns = "CarpetIOASCII"
IOASCII::one_file_per_group = yes
IOASCII::output_symmetry_points = no
IOASCII::out3D_ghosts = no
IOASCII::out0D_every = 128
IOASCII::out0D_vars = "
Carpet::timing
CarpetReduce::weight
ADMBase::metric
ADMBase::curv
ADMBase::lapse
ADMBase::shift
ADMBase::dtlapse
ADMBase::dtshift
WEYLSCAL4::Psi4r
WEYLSCAL4::Psi4i
ML_ADMConstraints::ML_Ham
ML_ADMConstraints::ML_mom
SphericalSurface::sf_active
SphericalSurface::sf_valid
SphericalSurface::sf_info
SphericalSurface::sf_radius
SphericalSurface::sf_origin
SphericalSurface::sf_coordinate_descriptors
QuasiLocalMeasures::qlm_state
QuasiLocalMeasures::qlm_grid_int
QuasiLocalMeasures::qlm_grid_real
QuasiLocalMeasures::qlm_scalars
QuasiLocalMeasures::qlm_multipole_moments
"
IOASCII::out1D_every = 128
IOASCII::out1D_vars = "
CarpetReduce::weight
ADMBase::metric
ADMBase::curv
ADMBase::lapse
ADMBase::shift
ADMBase::dtlapse
ADMBase::dtshift
WEYLSCAL4::Psi4r
WEYLSCAL4::Psi4i
ML_ADMConstraints::ML_Ham
ML_ADMConstraints::ML_mom
SphericalSurface::sf_radius
QuasiLocalMeasures::qlm_shapes
QuasiLocalMeasures::qlm_coordinates
QuasiLocalMeasures::qlm_tetrad_l
QuasiLocalMeasures::qlm_tetrad_n
QuasiLocalMeasures::qlm_tetrad_m
QuasiLocalMeasures::qlm_newman_penrose
QuasiLocalMeasures::qlm_weyl_scalars
QuasiLocalMeasures::qlm_ricci_scalars
QuasiLocalMeasures::qlm_twometric
QuasiLocalMeasures::qlm_killing_vector
QuasiLocalMeasures::qlm_killed_twometric
QuasiLocalMeasures::qlm_invariant_coordinates
QuasiLocalMeasures::qlm_3determinant
"
IOASCII::out2D_every = 128
IOASCII::out2D_vars = "
SphericalSurface::sf_radius
"
Activethorns = "CarpetIOHDF5"
IOHDF5::out_every = 512
IOHDF5::one_file_per_group = yes
IOHDF5::output_symmetry_points = no
IOHDF5::out3D_ghosts = no
IOHDF5::compression_level = 1
IOHDF5::use_checksums = yes
IOHDF5::out_vars = "
CarpetReduce::weight
ADMBase::metric
ADMBase::curv
ADMBase::lapse
ADMBase::shift
ADMBase::dtlapse
ADMBase::dtshift
WEYLSCAL4::Psi4r
WEYLSCAL4::Psi4i
ML_ADMConstraints::ML_Ham
ML_ADMConstraints::ML_mom
"
IOHDF5::checkpoint = no
IO::checkpoint_dir = $parfile
IO::checkpoint_ID = no
IO::checkpoint_every_walltime_hours = 6.0
IO::checkpoint_on_terminate = no
IO::recover = "autoprobe"
IO::recover_dir = $parfile
ActiveThorns = "Formaline"
ActiveThorns = "TimerReport"
TimerReport::out_every = 5120
TimerReport::out_filename = "TimerReport"
TimerReport::output_all_timers_together = yes
TimerReport::output_all_timers_readable = yes
TimerReport::n_top_timers = 20
More information about the Users
mailing list