Comment (by Roland Haas):

I gave ML_QC0.par a test run using 6 MPI ranks. I used either the Proca release code or Proca with Carpet set to rhaas/commstate_free_buffers instead.

With this I observe a change in allocated memory at the regrid in iteration 2048.

In the plot there are two fields of systemstatistics' output shown. max_rss which is the amount of memory currently claimed from the OS and which includes memory already free()ed but not yet returned to the OS. uordblks instead (see man mallinfo) is the amount of memory currently allocated via malloc (and new etc). The plot shows that applying the patch does (a) reduce the amount of memory malloc’ed on the rank with the largest such value decreases which is what should happen once one frees communication buffers and (b) that this apparently is enough to convince free() to return memory to the OS (drop in RSS). (b) is not something one can guarantee since memory fragmentation can prevent this since memory must be returned to the OS as hole pages (or worse as a whole chunk if mmap was not used to request it but setbrk).

I will try and see if there are differences i n ML_QC3 as well and maybe if there are differences late on in the ML_QC0 run as well.

I have not tested @Gabriele Bozzola 's files (yet) because they require an actual run with a number of nodes on Blue Waters.

@Miguel Zilhão do you see a drop in rss after it=2048 in your test run or no change at all? Note that rss is not really controllable by user code (since it depends on when free() returns memory to the OS) and that all one can really do is control uordblks by calling free().

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2194/memory-increase-during-regridding