Comment (by Ian Hinder):

Sorry I didn’t see this before it was closed. I would suggest that you first confirm that the problem is due to fragmentation. glibc malloc is fairly useless for this. I propose

1. Run a test case to confirm the problem, e.g. one of the cases that people have posted above. Wait until it actually crashes with OOM - don’t just look at reported memory statistics, especially not the RSS, which doesn’t tell you what you need to know.
2. Repeat the test with tcmalloc (see my instructions in a comment above). Look at the tcmalloc statistics. You should be able to determine the amount of “unused” memory (related to fragmentation). Ask me if you need help interpreting the tcmalloc statistics. Run it until it crashes - tcmalloc might help with fragmentation “out of the box”.

When I looked into this again in 2018 (I had previously tried to debug the problem for my own runs years before), the tentative conclusion did seem to point to fragmentation (see http://lists.einsteintoolkit.org/pipermail/users/2018-August/006434.html). But this was never fully confirmed. There were also hints that allocated memory was increasing, even though memory allocated for gridfunctions was not. So before concluding that fragmentation is the reason, it would be good to see a clear demonstration of this. If fragmentation is the reason, use of tcmalloc might reduce fragmentation, and there are parameters and strategies you can use to help.

My own solution to the problem was to run shorter jobs between checkpoints, so that the memory didn’t grow too high before the job checkpoint-terminated.

‌

--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2194/memory-increase-during-regridding