[Users] memory leak in Carpet?
Miguel Zilhão
miguel.zilhao.nogueira at tecnico.ulisboa.pt
Thu Aug 2 02:45:07 CDT 2018
hi all,
some more information regarding this. i've ran a simulation based on the provided parfile
qc0-mclachlan.par. i'm attaching a crude plot with the memory consumption, as reported by
systemstatistics-process_memory and carpet-memory_procs, as function of the iteration.
the jump at iteration ~2048 corresponds to the time where some non-trivial regridding occurred. if
i'm interpreting the plot correctly, while the total memory consumption of the system (as reported
by systemstatistics) increases, the memory that carpet reports to be using is actually decreasing.
is this a sign that something is probably leaking memory?
in case it's useful, i'm also attaching the exact parameter file i've used.
thanks,
Miguel
On 29/07/2018 21:12, Miguel Zilhão wrote:
> hi Ian,
>
> thanks again for your thorough reply! i've checked the low resolution run that i
> have (and which ran successfully) and the pattern that i observe is that maxrss
> typically grows after each regridding. the increase in maxrss is not monotonic,
> though, as it does go down at times. then, after the BHs merge and the
> regridding stops, maxrss settles down (to a value still a bit higher than the
> maxrss at t=0). this is all for the first segment of the simulation, as i
> haven't done any checkpointing for this run.
>
> so i guess i don't have enough data to conclude whether what i'm observing is
> indeed a memory leak, or if it's just Carpet's regridding algorithm doing what
> it's supposed to do. in any case, i guess what surprises me is that, at times,
> the memory consumption can be bigger than what it was at the beginning of the
> run by a factor of 1.6, which can easily lead to an out-of-memory situation...
>
> unfortunately these days i don't have the time to investigate this any
> further...
>
> many thanks,
> Miguel
>
> On 23/07/2018 16:06, ian.hinder at aei.mpg.de wrote:
>>
>>
>>> On 23 Jul 2018, at 15:00, Miguel Zilhão <miguel.zilhao.nogueira at tecnico.ulisboa.pt
>>> <mailto:miguel.zilhao.nogueira at tecnico.ulisboa.pt>> wrote:
>>>
>>> hi Ian and all,
>>>
>>>>>> This could be caused by memory fragmentation due to all the freeing and mallocing that happens
>>>>>> during regridding when the sizes of the grids change. Can you try using tcmalloc or jemalloc
>>>>>> instead of glibc malloc and reporting back? One workaround could be to run shorter simulations
>>>>>> (i.e. set a walltime of 12 h instead of 24 h).
>>>>>
>>>>> thanks for your reply. in one of my cases, for the resolution used and the available memory, i
>>>>> was out of memory quite quickly -- within 6 hours or so... so unfortunately it becomes a bit
>>>>> impractical for large simulations...
>>>>>
>>>>> what would i need to do in order to use tcmalloc or jemalloc?
>>>> I have used tcmalloc. I think you will need the following:
>>>> - Install tcmalloc (https://github.com/gperftools/gperftools), and libunwind, which it depends on.
>>>> - In your optionlist, link with tcmalloc. I have
>>>> LDFLAGS = -rdynamic -L/home/ianhin/software/gperftools-2.1/lib
>>>> -Wl,-rpath,/home/ianhin/software/gperftools-2.1/lib -ltcmalloc
>>>> This should be sufficient I think for tcmalloc to be used instead of glibc malloc. Try this out,
>>>> and see if things are better. I also have a thorn which hooks into the tcmalloc API. You can
>>>> get it from
>>>
>>> thanks a lot for these pointers. i've tried it out, though i've used tcmalloc from ubuntu's
>>> repositories and therefore compiled ET with -ltcmalloc_minimal. i don't know whether this makes a
>>> difference, but from the trial run that i'm doing i so far seem to see the same memory increase i
>>> had seen before...
>>
>> Hi,
>>
>> Did you use the tcmalloc thorn and the parameters to make it release memory back to the OS after
>> each regridding?
>>
>>> is there anything else that can be tried to try to pinpoint this issue? it seems to be serious...
>>> i looked for an open ticket but didn't find anything. shall i submit one?
>>
>> You can look at the Carpet::memory_procs variable (in Carpet/Carpet/interface.ccl). This will tell
>> you how much memory Carpet has allocated for different things. If one of these is growing during
>> the run, but resets to a lower value after checkpoint recovery, then that suggests a memory leak.
>>
>> Hmm. Now this is coming back to me. I just searched my email, and I found a draft email that I
>> never sent from 2015 with subject "Memory leak in Carpet?". Here it is:
>>
>>> Does anyone have any reason to suspect that something in Carpet might be leaking memory? I have a
>>> fairly straightforward QC0 simulation, based on qc0-mclachlan, and it looks like something is
>>> leaking memory.
>>>
>>> I have looked at several diagnostics.
>>>
>>> – carpet::grid_functions: this contains the amount of memory Carpet thinks is allocated for grid
>>> functions. Since I have regridding, this is not in general going to be constant, but it turns out
>>> that it remains approximate constant at about 5 GB per process (average and maximum across
>>> processes are about the same). Each process has 12 GB available on Datura. So I should be well
>>> within the memory limits of the machine, by more than a factor of 2.
>>>
>>> – The run crashes with signal 9 during regridding at some point; this is probably the OOM killer.
>>> – SystemStatistics::swap_used_mb starts to grow after the first regridding, and seems to grow
>>> linearly throughout the run. The crash time corresponds to it hitting about 18 GB, which is the
>>> maximum swap configured on the node.
>>>
>>> – SystemStatistics::arena: This is the 'arena' field from mallinfo
>>> (http://man7.org/linux/man-pages/man3/mallinfo.3.html) which is supposed to give 'Non-mmapped
>>> space allocated (bytes)'. This suffers from being stored in a 32 bit integer in the mallinfo
>>> structure (https://sourceware.org/ml/libc-alpha/2014-11/msg00431.html), but I have adjusted it
>>> manually by adding 4 GB when it looks like it is dropping unphysically. This shows that the
>>> amount of non-mmapped memory allocated is increasing on the order of 1 GB on each regridding, and
>>> not between regriddings. Firstly, the amount of memory allocated shouldn't be increasing so much,
>>> since carpet::grid_functions remains approximately constant. Secondly, I thought that we were
>>> supposed to be using mmap for grid function data, so why do we have such a large amount of
>>> non-mmapped memory?
>>>
>>> – SystemStatistics::hblkhd: This is the hblkhd field from mallinfo, which is supposed to be "Space
>>> allocated in mmapped regions (bytes)". This increases to about 2 GB after a couple of
>>> regriddings, but then stays roughly constant at 2 GB, which seems fine, apart from the fact that I
>>> had hoped that mmap was being used for all the gridfunction data
>>
>> I suspect I got distracted doing my own investigations while I was writing it, and then lost track
>> of it. Typically, I think I run with excess memory for each run, so that by 24 h, it hasn't grown
>> too badly.
>>
>> Further searching finds an email from someone else saying that they had memory leak problems and
>> asking me about it. My reply was:
>>
>> On 5 May 2015, at 17:14, Ian Hinder <ian.hinder at aei.mpg.de <mailto:ian.hinder at aei.mpg.de>> wrote:
>>
>>> Hi,
>>>
>>> I have added Erik and Roland to CC, as we have been discussing this; I hope this is OK.
>>>
>>> It sounds very similar. I am running on several hundred cores (<600) and the simulations often
>>> fail with OOM errors after less than a day. First the RSS grows, then the swap, then the
>>> OOM-killer kills it. I have observed this both on Datura and Hydra. Stopping the simulations and
>>> recovering from a checkpoint usually fixes the problem, and it runs on for another half day or so.
>>> I have done a fair amount of work on this, so I will summarise here.
>>>
>>> *Monitor process RSS*
>>>
>>> The process resident set size is the amount of address space which is currently mapped into
>>> physical memory by the OS. Thorn SystemStatistics can be used to measure this and put it into a
>>> Cactus variable, which can then be reduced across processes. I use:
>>>
>>> IOBasic::outInfo_every = 1
>>> IOBasic::outInfo_reductions = "maximum"
>>> IOBasic::outInfo_vars = "
>>> SystemStatistics::maxrss_mb
>>> SystemStatistics::swap_used_mb
>>> Carpet::gridfunctions
>>> "
>>>
>>> and
>>>
>>> IOScalar::outScalar_every = 128
>>> IOScalar::outScalar_vars = "
>>> SystemStatistics::process_memory_mb
>>> Carpet::memory_procs
>>> "
>>>
>>> SystemStatistics calls its variable "maxrss" but it should actually be called "rss", as that is
>>> what is output. maxrss is also available from the OS, and would give the maximum the RSS had ever
>>> been during the process lifetime.
>>>
>>> Carpet::gridfunctions (in Carpet::memory_procs) measures the amount of memory Carpet has allocated
>>> in gridfunctions. For me, this remains essentially flat, whereas maxrss grows after each
>>> regridding until it reaches the maximum available, then the swap starts to grow. This indicates
>>> that the problem is not due to Carpet allocating more and more grid points due to grids changing
>>> size. It could be due to failing to free allocated memory (a leak) or freed data taking up space
>>> which cannot be used for further allocations or returned to the OS (fragmentation).
>>>
>>> *Terminate and checkpoint on OOM*
>>>
>>> I have a local change to SystemStatistics which adds parameters for maximum values of RSS and swap
>>> usage, above which it calls CCTK_TerminateNext, so if you have checkpoint_on_terminate, you get a
>>> clean termination and can continue the run without losing too much CPU time. I have been running
>>> with this for a couple of weeks now, and it works as advertised. I have a branch with this on,
>>> but I just realised it conflicts with a change Erik made. If you want this, let me know and I
>>> will sort it out.
>>>
>>> *Memory profiling*
>>>
>>> The malloc implementation in glibc provides no usable statistics. mallinfo is limited to 32 bit
>>> integers, which overflow for 64 bit systems. malloc_info, at least in the version on datura,
>>> doesn't include memory allocated via mmap. Useless. Instead, you need to use an external memory
>>> profiler to see what is going on. I have used "igprof" successfully, and this shows me that there
>>> is no "leak" of allocated memory corresponding to the increase in RSS. i.e. the problem is not
>>> caused by forgetting to free something. This suggests that the problem is fragmentation, where
>>> malloc has unallocated blocks of memory which it does not or cannot return to the OS. Malloc
>>> allocates memory in two ways: either in its main heap, or by allocating anonymous mmap regions. I
>>> had thought that only the latter could be fully returned to the OS, but this is not true. Any
>>> region of address space can be marked as unused (internally via the madvise(MADV_DONTNEED) system
>>> call) and a malloc implementation can do this on regions of its address space which have been
>>> freed. If such regions are too small (smaller than a page), then they could accumulate and not be
>>> returned to the OS.
>>>
>>> *Alternative malloc implementations*
>>>
>>> At the suggestion of Roland, I tried using the tcmalloc
>>> (http://gperftools.googlecode.com/git/doc/tcmalloc.html) library, which is a drop-in replacement
>>> for glibc malloc which is part of gperftools. This works fairly easily. You can compile the
>>> "minimal" version with no dependencies and then modify your optionlist:
>>>
>>> CPPFLAGS = -I/home/rhaas/software/gperftools-2.1/include/gperftools
>>> LDFLAGS = -L/home/rhaas/software/gperftools-2.1/lib
>>> -Wl,-rpath,/home/rhaas/software/gperftools-2.1/lib -ltcmalloc_minimal
>>>
>>> I found in one example case that this reduced the RSS process growth, so I am now using it for all
>>> my simulations. However, I still run into the same problem eventually, so it might be that it
>>> makes it better but doesn't solve it completely.
>>>
>>> tcmalloc has an introspection interface which is presumably not as useless as glibc's malloc:
>>> http://gperftools.googlecode.com/git/doc/tcmalloc.html. I haven't tried this yet.
>>>
>>> *Checkpoint recovery*
>>>
>>> I noticed from the igprof profile that there are 11000 allocations (and frees) during checkpoint
>>> recovery on one process, all from the HDF5 uncompression routine. This is the "deflate" filter.
>>> When it decompresses a dataset, it allocates a buffer, initially sized the same as the compressed
>>> dataset (really dumb, as it will always need to be bigger). It then uncompresses into the buffer,
>>> "realloc"ing the buffer to twice the size each time it runs out of space. You can imagine that
>>> this might cause a lot of fragmentation. There is no tunable parameter, but we could modify the
>>> code (it's in https://svn.hdfgroup.uiuc.edu/hdf5/tags/hdf5-1_8_12/src/H5Zdeflate.c) to use a much
>>> larger starting buffer size, in the hope that this reduces the number of reallocs, and hence the
>>> amount of fragmentation. This wouldn't help the accumulated RSS, but it would probably produce a
>>> one-off decrease in the amount of fragmentation. I am currently not using periodic checkpointing,
>>> so I don't know if the compression routine has the same problem. Probably not, since it knows the
>>> output buffer size has to be smaller than the input buffer size. Apparently Frank Löffler also
>>> modified this routine, which solved some of his problems of running out of memory during recovery.
>>> Another alternative would be to disable checkpoint compression.
>>>
>>> To see if you are suffering from the same problem, I think the quickest way would be to link
>>> against tcmalloc and use MallocExtension::instance()->GetNumericProperty(property_name, value)
>>> from tcmalloc to read off the generic.current_allocated_bytes property (Number of bytes used by
>>> the application. This will not typically match the memory use reported by the OS, because it does
>>> not include TCMalloc overhead or memory fragmentation). You could also look at the other
>>> properties they provide. Then compare this with the process RSS from systemstatistics, and
>>> Carpet's gridfunctions variable, and check to see if you actually have a memory leak, or if you
>>> are suffering from fragmentation.
>>
>> This brings to mind another question: are you using HDF5 compression? If so, do you see the same
>> problem if you switch it off?
>>
>> And finally: do you get this growth on the very first segment of a simulation, or only on subsequent
>> segments? I am thinking that checkpoint *recovery* severely fragments the heap, especially with
>> compression, and this somehow causes growth in the RSS with each regridding.
>>
>> So, while I had forgotten about all this, it turns out that I had actually thought quite a lot about
>> it :)
>>
>> --
>> Ian Hinder
>> https://ianhinder.net
>>
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: QC0_mem.pdf
Type: application/pdf
Size: 11526 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20180802/e52d9790/attachment-0001.pdf
-------------- next part --------------
#------------------------------------------------------------------------------
ActiveThorns = "
ADMBase
ADMCoupling
ADMMacros
AEILocalInterp
AHFinderDirect
Boundary
Carpet
CarpetInterp
CarpetIOASCII
CarpetIOBasic
CarpetIOHDF5
CarpetIOScalar
CarpetLib
CarpetMask
CarpetReduce
CarpetRegrid2
CarpetSlab
CarpetTracker
CartGrid3D
CoordBase
CoordGauge
Dissipation
# Formaline
Fortran
GenericFD
GSL
HDF5
InitBase
IOUtil
LocalInterp
LoopControl
ML_ADMConstraints
ML_BSSN
ML_BSSN_Helper
MoL
Multipole
NaNChecker
NewRad
# PunctureTracker
QuasiLocalMeasures
ReflectionSymmetry
RotatingSymmetry180
Slab
SpaceMask
SphericalSurface
StaticConformal
SummationByParts
SymBase
SystemStatistics
TerminationTrigger
Time
TimerReport
TmunuBase
TwoPunctures
Vectors
WeylScal4
"
#------------------------------------------------------------------------------
# Grid setup
#------------------------------------------------------------------------------
CartGrid3D::type = "coordbase"
Carpet::domain_from_coordbase = yes
CoordBase::domainsize = "minmax"
# make sure all (xmax - xmin)/dx are integers!
CoordBase::xmin = 0.00
CoordBase::ymin = -120.00
CoordBase::zmin = 0.00
CoordBase::xmax = +120.00
CoordBase::ymax = +120.00
CoordBase::zmax = +120.00
CoordBase::dx = 2.00
CoordBase::dy = 2.00
CoordBase::dz = 2.00
driver::ghost_size = 3
CoordBase::boundary_size_x_lower = 3
CoordBase::boundary_size_y_lower = 3
CoordBase::boundary_size_z_lower = 3
CoordBase::boundary_size_x_upper = 3
CoordBase::boundary_size_y_upper = 3
CoordBase::boundary_size_z_upper = 3
CoordBase::boundary_shiftout_x_lower = 1
CoordBase::boundary_shiftout_y_lower = 0
CoordBase::boundary_shiftout_z_lower = 1
CarpetRegrid2::symmetry_rotating180 = yes
ReflectionSymmetry::reflection_x = no
ReflectionSymmetry::reflection_y = no
ReflectionSymmetry::reflection_z = yes
ReflectionSymmetry::avoid_origin_x = yes
ReflectionSymmetry::avoid_origin_y = yes
ReflectionSymmetry::avoid_origin_z = no
# Mesh refinement
#------------------------------------------------------------------------------
Carpet::max_refinement_levels = 7
CarpetRegrid2::num_centres = 3
CarpetRegrid2::num_levels_1 = 7
CarpetRegrid2::position_x_1 = +1.168642873
CarpetRegrid2::radius_1[ 1] = 64.0
CarpetRegrid2::radius_1[ 2] = 16.0
CarpetRegrid2::radius_1[ 3] = 8.0
CarpetRegrid2::radius_1[ 4] = 4.0
CarpetRegrid2::radius_1[ 5] = 2.0
CarpetRegrid2::radius_1[ 6] = 1.0
CarpetRegrid2::movement_threshold_1 = 0.16
CarpetRegrid2::num_levels_2 = 7
CarpetRegrid2::position_x_2 = -1.168642873
CarpetRegrid2::radius_2[ 1] = 64.0
CarpetRegrid2::radius_2[ 2] = 16.0
CarpetRegrid2::radius_2[ 3] = 8.0
CarpetRegrid2::radius_2[ 4] = 4.0
CarpetRegrid2::radius_2[ 5] = 2.0
CarpetRegrid2::radius_2[ 6] = 1.0
CarpetRegrid2::movement_threshold_2 = 0.16
CarpetRegrid2::num_levels_3 = 7
CarpetRegrid2::active_3 = no
CarpetRegrid2::radius_3[ 1] = 64.0
CarpetRegrid2::radius_3[ 2] = 16.0
CarpetRegrid2::radius_3[ 3] = 8.0
CarpetRegrid2::radius_3[ 4] = 4.0
CarpetRegrid2::radius_3[ 5] = 2.0
CarpetRegrid2::radius_3[ 6] = 1.0
CarpetRegrid2::movement_threshold_3 = 0.16
Carpet::use_buffer_zones = yes
Carpet::prolongation_order_space = 5
Carpet::prolongation_order_time = 2
CarpetRegrid2::freeze_unaligned_levels = yes
CarpetRegrid2::regrid_every = 128
CarpetRegrid2::verbose = no
Carpet::grid_structure_filename = "carpet-grid-structure"
Carpet::grid_coordinates_filename = "carpet-grid-coordinates"
Carpet::time_refinement_factors = "[1, 1, 2, 4, 8, 16, 32, 64, 128, 256]"
Time::dtfac = 0.25
# Initial Data
#------------------------------------------------------------------------------
ADMBase::metric_type = "physical"
ADMBase::initial_data = "twopunctures"
ADMBase::initial_lapse = "twopunctures-averaged"
ADMBase::initial_shift = "zero"
ADMBase::initial_dtlapse = "zero"
ADMBase::initial_dtshift = "zero"
ADMBase::lapse_timelevels = 3
ADMBase::shift_timelevels = 3
ADMBase::metric_timelevels = 3
# TwoPunctures::target_M_plus = 0.5
# TwoPunctures::target_M_minus = 0.5
TwoPunctures::par_m_plus = 0.453
TwoPunctures::par_m_minus = 0.453
TwoPunctures::par_b = 1.168642873
TwoPunctures::center_offset[0] = 0
TwoPunctures::par_P_plus[0] = 0.0
TwoPunctures::par_P_plus[1] = 0.3331917498
TwoPunctures::par_P_plus[2] = 0.
TwoPunctures::par_P_minus[0] = 0.0
TwoPunctures::par_P_minus[1] = -0.3331917498
TwoPunctures::par_P_minus[2] = 0.
TwoPunctures::par_S_plus[0] = 0.
TwoPunctures::par_S_plus[1] = 0.
TwoPunctures::par_S_plus[2] = 0.
TwoPunctures::par_S_minus[0] = 0.
TwoPunctures::par_S_minus[1] = 0.
TwoPunctures::par_S_minus[2] = 0.
# TwoPunctures::give_bare_mass = yes
TwoPunctures::TP_epsilon = 1.0e-2
TwoPunctures::TP_Tiny = 1.0e-2
TwoPunctures::npoints_A = 30
TwoPunctures::npoints_B = 30
TwoPunctures::npoints_phi = 16
TwoPunctures::grid_setup_method = "evaluation"
TwoPunctures::keep_u_around = yes
TwoPunctures::verbose = yes
InitBase::initial_data_setup_method = "init_some_levels"
Carpet::init_fill_timelevels = yes
Carpet::init_3_timelevels = no
# Evolution
#------------------------------------------------------------------------------
ADMBase::evolution_method = "ML_BSSN"
ADMBase::lapse_evolution_method = "ML_BSSN"
ADMBase::shift_evolution_method = "ML_BSSN"
ADMBase::dtlapse_evolution_method = "ML_BSSN"
ADMBase::dtshift_evolution_method = "ML_BSSN"
ML_BSSN::harmonicN = 1 # 1+log
ML_BSSN::harmonicF = 2.0 # 1+log
ML_BSSN::ShiftGammaCoeff = 0.75
ML_BSSN::BetaDriver = 1.0
ML_BSSN::advectLapse = 1
ML_BSSN::advectShift = 1
ML_BSSN::MinimumLapse = 1.0e-8
ML_BSSN::conformalMethod = 1 # 1 for W
# Boundary conditions
#------------------------------------------------------------------------------
ML_BSSN::initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::rhs_boundary_condition = "NewRad"
Boundary::radpower = 2
ML_BSSN::ML_log_confac_bound = "none"
ML_BSSN::ML_metric_bound = "none"
ML_BSSN::ML_Gamma_bound = "none"
ML_BSSN::ML_trace_curv_bound = "none"
ML_BSSN::ML_curv_bound = "none"
ML_BSSN::ML_lapse_bound = "none"
ML_BSSN::ML_dtlapse_bound = "none"
ML_BSSN::ML_shift_bound = "none"
ML_BSSN::ML_dtshift_bound = "none"
# Spatial finite differencing
#------------------------------------------------------------------------------
SummationByParts::order = 4
ML_BSSN::fdOrder = 4
Dissipation::order = 5
Dissipation::vars = "
ML_BSSN::ML_log_confac
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
"
# Integration method
#------------------------------------------------------------------------------
MoL::ODE_Method = "RK4"
MoL::MoL_Intermediate_Steps = 4
MoL::MoL_Num_Scratch_Levels = 1
Carpet::num_integrator_substeps = 4
# Spherical surfaces
#------------------------------------------------------------------------------
SphericalSurface::nsurfaces = 3
SphericalSurface::maxntheta = 66
SphericalSurface::maxnphi = 124
SphericalSurface::verbose = no
# Horizon 1
SphericalSurface::ntheta [0] = 41
SphericalSurface::nphi [0] = 80
SphericalSurface::nghoststheta [0] = 2
SphericalSurface::nghostsphi [0] = 2
CarpetMask::excluded_surface [0] = 0
CarpetMask::excluded_surface_factor [0] = 1.0
# Horizon 2
SphericalSurface::ntheta [1] = 41
SphericalSurface::nphi [1] = 80
SphericalSurface::nghoststheta [1] = 2
SphericalSurface::nghostsphi [1] = 2
CarpetMask::excluded_surface [1] = 1
CarpetMask::excluded_surface_factor [1] = 1.0
# Common horizon
SphericalSurface::ntheta [2] = 41
SphericalSurface::nphi [2] = 80
SphericalSurface::nghoststheta [2] = 2
SphericalSurface::nghostsphi [2] = 2
CarpetMask::excluded_surface [2] = 2
CarpetMask::excluded_surface_factor [2] = 1.0
CarpetMask::verbose = no
# Puncture tracking
#------------------------------------------------------------------------------
CarpetTracker::surface[0] = 0
CarpetTracker::surface[1] = 1
CarpetTracker::surface[2] = 2
# PunctureTracker::track [0] = yes
# PunctureTracker::initial_x [0] = 3
# PunctureTracker::which_surface_to_store_info[0] = 0
# PunctureTracker::track [1] = yes
# PunctureTracker::initial_x [1] = -3
# PunctureTracker::which_surface_to_store_info[1] = 1
# PunctureTracker::verbose = no
# Wave extraction
#------------------------------------------------------------------------------
WeylScal4::fd_order = "4th"
WeylScal4::offset = 1e-8
Multipole::nradii = 4
Multipole::radius[0] = 30
Multipole::radius[1] = 50
Multipole::radius[2] = 80
Multipole::radius[3] = 100
Multipole::ntheta = 120
Multipole::nphi = 240
Multipole::variables = "
WeylScal4::Psi4r{sw=-2 cmplx='WeylScal4::Psi4i' name='Psi4'}
"
# Multipole::integration_method = "Simpson"
Multipole::out_every = 64
Multipole::l_max = 4
Multipole::output_hdf5 = no
Multipole::output_ascii = yes
# Horizons
#------------------------------------------------------------------------------
# AHFinderDirect::verbose_level = "algorithm highlights"
AHFinderDirect::verbose_level = "physics details"
AHFinderDirect::output_BH_diagnostics = "true"
AHFinderDirect::run_at_CCTK_POST_RECOVER_VARIABLES = no
AHFinderDirect::N_horizons = 3
AHFinderDirect::find_every = 128
AHFinderDirect::output_h_every = 0
AHFinderDirect::max_Newton_iterations__initial = 50
AHFinderDirect::max_Newton_iterations__subsequent = 50
AHFinderDirect::max_allowable_Theta_growth_iterations = 10
AHFinderDirect::max_allowable_Theta_nonshrink_iterations = 10
AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::geometry_interpolator_pars = "order=4"
AHFinderDirect::surface_interpolator_name = "Lagrange polynomial interpolation"
AHFinderDirect::surface_interpolator_pars = "order=4"
AHFinderDirect::move_origins = yes
AHFinderDirect::reshape_while_moving = yes
AHFinderDirect::predict_origin_movement = yes
AHFinderDirect::origin_x [1] = +1.168642873
AHFinderDirect::initial_guess__coord_sphere__x_center[1] = +1.168642873
AHFinderDirect::initial_guess__coord_sphere__radius [1] = 0.25
AHFinderDirect::which_surface_to_store_info [1] = 0
AHFinderDirect::set_mask_for_individual_horizon [1] = no
AHFinderDirect::reset_horizon_after_not_finding [1] = no
# AHFinderDirect::track_origin_from_grid_scalar [1] = yes
# AHFinderDirect::track_origin_source_x [1] = "PunctureTracker::pt_loc_x[0]"
# AHFinderDirect::track_origin_source_y [1] = "PunctureTracker::pt_loc_y[0]"
# AHFinderDirect::track_origin_source_z [1] = "PunctureTracker::pt_loc_z[0]"
AHFinderDirect::max_allowable_horizon_radius [1] = 3
AHFinderDirect::dont_find_after_individual_time [1] = 30.0
AHFinderDirect::origin_x [2] = -1.168642873
AHFinderDirect::initial_guess__coord_sphere__x_center[2] = -1.168642873
AHFinderDirect::initial_guess__coord_sphere__radius [2] = 0.25
AHFinderDirect::which_surface_to_store_info [2] = 1
AHFinderDirect::set_mask_for_individual_horizon [2] = no
AHFinderDirect::reset_horizon_after_not_finding [2] = no
# AHFinderDirect::track_origin_from_grid_scalar [2] = yes
# AHFinderDirect::track_origin_source_x [2] = "PunctureTracker::pt_loc_x[1]"
# AHFinderDirect::track_origin_source_y [2] = "PunctureTracker::pt_loc_y[1]"
# AHFinderDirect::track_origin_source_z [2] = "PunctureTracker::pt_loc_z[1]"
AHFinderDirect::max_allowable_horizon_radius [2] = 3
AHFinderDirect::dont_find_after_individual_time [2] = 30.0
AHFinderDirect::origin_x [3] = 0
AHFinderDirect::find_after_individual_time [3] = 15.0
AHFinderDirect::initial_guess__coord_sphere__x_center[3] = 0
AHFinderDirect::initial_guess__coord_sphere__radius [3] = 1.0
AHFinderDirect::which_surface_to_store_info [3] = 2
AHFinderDirect::set_mask_for_individual_horizon [3] = no
AHFinderDirect::max_allowable_horizon_radius [3] = 6
# Isolated Horizons
#-------------------------------------------------------------------------------
QuasiLocalMeasures::verbose = yes
QuasiLocalMeasures::veryverbose = no
QuasiLocalMeasures::interpolator = "Lagrange polynomial interpolation"
QuasiLocalMeasures::interpolator_options = "order=4"
QuasiLocalMeasures::spatial_order = 4
QuasiLocalMeasures::num_surfaces = 3
QuasiLocalMeasures::surface_index [0] = 0
QuasiLocalMeasures::surface_index [1] = 1
QuasiLocalMeasures::surface_index [2] = 2
# Check for NaNs
#-------------------------------------------------------------------------------
Carpet::poison_new_timelevels = yes
CarpetLib::poison_new_memory = yes
Carpet::check_for_poison = no
NaNChecker::check_every = 512
NanChecker::check_after = 0
NaNChecker::report_max = 10
# NaNChecker::verbose = "all"
NaNChecker::action_if_found = "terminate"
NaNChecker::out_NaNmask = yes
NaNChecker::check_vars = "
ML_BSSN::ML_log_confac
"
# Timers
#-------------------------------------------------------------------------------
Cactus::cctk_timer_output = "full"
TimerReport::out_every = 5120
TimerReport::n_top_timers = 40
TimerReport::output_all_timers_together = yes
TimerReport::output_all_timers_readable = yes
TimerReport::output_schedule_timers = no
# I/O thorns
#-------------------------------------------------------------------------------
Cactus::cctk_run_title = $parfile
IO::out_dir = $parfile
IOScalar::one_file_per_group = yes
IOASCII::one_file_per_group = yes
IOHDF5::use_checksums = no
IOHDF5::one_file_per_group = no
IOBasic::outInfo_every = 4
IOBasic::outInfo_reductions = "minimum maximum"
IOBasic::outInfo_vars = "
Carpet::physical_time_per_hour
ML_BSSN::ML_log_confac
SystemStatistics::maxrss_mb
"
# for scalar reductions of 3D grid functions
IOScalar::outScalar_every = 64
IOScalar::outScalar_reductions = "minimum maximum average norm2"
IOScalar::outScalar_vars = "
ML_BSSN::ML_Ham
ML_BSSN::ML_mom
SystemStatistics::maxrss_mb
SystemStatistics::process_memory_mb
Carpet::memory_procs
"
# output just at one point (0D)
IOASCII::out0D_every = 32
IOASCII::out0D_vars = "
Carpet::timing
# PunctureTracker::pt_loc
QuasiLocalMeasures::qlm_scalars{out_every = 128}
"
IOASCII::output_symmetry_points = no
IOASCII::out3D_ghosts = no
# 1D text output
IOASCII::out1D_every = 128
IOASCII::out1D_d = no
IOASCII::out1D_x = yes
IOASCII::out1D_y = no
IOASCII::out1D_z = no
IOASCII::out1D_vars = "
ADMBase::lapse
ML_BSSN::ML_Ham
ML_BSSN::ML_mom
"
# 1D HDF5 output
#IOHDF5::out1D_every = 256
#IOHDF5::out1D_d = no
#IOHDF5::out1D_x = yes
#IOHDF5::out1D_y = no
#IOHDF5::out1D_z = no
#IOHDF5::out1D_vars = "
# ADMBase::lapse
#"
# # 2D HDF5 output
# IOHDF5::out2D_every = 256
# IOHDF5::out2D_xy = yes
# IOHDF5::out2D_xz = no
# IOHDF5::out2D_yz = no
# IOHDF5::out2D_vars = "
# ADMBase::lapse
# "
# # 3D HDF5 output
# IOHDF5::out_every = 8192
# IOHDF5::out_vars = "
# ADMBase::lapse
# "
Carpet::verbose = no
Carpet::veryverbose = no
Carpet::schedule_barriers = no
Carpet::storage_verbose = no
CarpetLib::output_bboxes = no
Cactus::cctk_full_warnings = yes
Cactus::highlight_warning_messages = no
# Checkpointing and recovery
#-------------------------------------------------------------------------------
CarpetIOHDF5::checkpoint = yes
IO::checkpoint_dir = "../checkpoints"
IO::checkpoint_ID = no
IO::checkpoint_every_walltime_hours = 23
IO::checkpoint_on_terminate = yes
IO::out_proc_every = 2
IO::checkpoint_keep = 1
IO::recover = "autoprobe"
IO::recover_dir = "../checkpoints"
IO::abort_on_io_errors = yes
CarpetIOHDF5::open_one_input_file_at_a_time = yes
CarpetIOHDF5::compression_level = 9
# Run termination
#-------------------------------------------------------------------------------
TerminationTrigger::max_walltime = 72 # hours
TerminationTrigger::on_remaining_walltime = 60 # minutes
TerminationTrigger::output_remtime_every_minutes = 60
TerminationTrigger::termination_from_file = yes
TerminationTrigger::termination_file = "terminate.txt"
TerminationTrigger::create_termination_file = yes
Cactus::terminate = "time"
Cactus::cctk_final_time = 300.0
More information about the Users
mailing list