hi Ian,
many thanks for your analysis. for that particular run there is no significant change in the memory usage for longer times. however, i've now rerun the configuration that had originally given me problems (with lower resolution so that i'd have enough memory) with tcmalloc and i've plotted those same curves. here, the memory used increases further. i'm attaching the plot, together with the parfile and stdout output in case it's useful.
thanks, Miguel
On 04/08/2018 20:01, ian.hinder@aei.mpg.de wrote:
On 3 Aug 2018, at 09:49, Miguel Zilhão <miguel.zilhao.nogueira@tecnico.ulisboa.pt mailto:miguel.zilhao.nogueira@tecnico.ulisboa.pt> wrote:
hi Ian,
sorry, i thought it was enough to see the qualitative trend of the curves... here's what i hope to be a better plot, with all the information. also, i have no hdf5 output for this run. i'm only saving plain text.
Hi Miguel,
So, from this, we can see:
- Carpet is using less memory for gridfunctions after the regridding than before. I suppose this
could happen if the BHs get closer to each other, and the coarser enclosed grids shrink. I'm a little surprised to see this on the very first regridding, but it's not a big effect anyway.
- The amount of *allocated* memory in the tcmalloc heap increases a bit after regridding. I don't
know why this would be, since the gridfunction memory should dominate, and this decreases. It would be interesting to see this plot for longer times. If the green curve continues to step up by about 80 MB every 2048 iterations, this could be the reason for running out of memory.
- The heap size increases by much more than the additional allocated memory. This suggests that the
heap has become fragmented, or that tcmalloc has not attempted to return memory to the OS. The tcmalloc thorn calls tcmalloc to release all memory to the OS in POSTREGRID, so as far as tcmalloc is concerned, no more memory can be returned. This suggests fragmentation; the free blocks are mixed up with allocated blocks so that entire pages cannot be mapped out. Can you set tcmalloc::report_every = 2048? The outputs a short summary of the heap status to stdout. I would again be interested to see whether this continues with a longer run. i.e. whether heap_size - allocated continues to increase.
- Interestingly, pageheap_unmapped grows a lot.
-- Ian Hinder https://ianhinder.net