On 19 Jul 2018, at 11:58, Miguel Zilhão <miguel.zilhao.nogueira@tecnico.ulisboa.pt> wrote:

hi all,

i've noticed that my runs (using latest ET release) with CarpetRegrid2 exhibit a significant
increase in memory during runtime. this seems to happen immediately after some non-trivial
regridding operation is done. the increase is steady, and at some point i run out of memory and the
simulation crashes. this is happening both on my workstation (running Ubuntu 18.04) as well as our
local cluster (running Debian 9). i was wondering if someone has seen something like this?

i have not seen this happen for simulations without CarpetRegrid2. i show below some relevant
portions of the stdout file for a standard inspiral BH run (note the last column--maxrss_mb):

This could be caused by memory fragmentation due to all the freeing and mallocing that happens during regridding when the sizes of the grids change. Can you try using tcmalloc or jemalloc instead of glibc malloc and reporting back? One workaround could be to run shorter simulations (i.e. set a walltime of 12 h instead of 24 h).