<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 5 Aug 2018, at 15:52, Miguel Zilhão <<a href="mailto:miguel.zilhao.nogueira@tecnico.ulisboa.pt" class="">miguel.zilhao.nogueira@tecnico.ulisboa.pt</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">hi Ian,<br class=""><br class="">many thanks for your analysis. for that particular run there is no significant change in the memory usage for longer times. however, i've now rerun the configuration that had originally given me problems (with lower resolution so that i'd have enough memory) with tcmalloc and i've plotted those same curves. here, the memory used increases further. i'm attaching the plot, together with the parfile and stdout output in case it's useful.<br class=""></div></div></blockquote><div><br class=""></div><div>Hi Miguel,</div><div><br class=""></div><div>The memory seems to reach a steady state by iteration ~3000. Can you run an example where it dies with an OOM?</div><div><br class=""></div><div>The meaning of the different tcmalloc statistics is described at <a href="https://gperftools.github.io/gperftools/tcmalloc.html" class="">https://gperftools.github.io/gperftools/tcmalloc.html</a> under "Generic Tcmalloc Status".</div><div><br class=""></div><div>From what I see here, the amount of memory allocated by Cactus grows then reaches a steady state (the green curve). This is not accounted for in the gridfunctions, so it must be from something else. I don't know what it might be. Does this run also have HDF5 output disabled?</div><div><br class=""></div><div>It looks like the allocated memory plus the unmapped memory would roughly equal the rss. That is a little surprising, since I would have expected the rss to exclude unmapped pages. Maybe until another process needs it, the kernel doesn't actually unmap it, for performance reasons (mapping it back again would cause a page fault).</div><div><br class=""></div><div>Can you check this by plotting tcmalloc::generic_current_allocated + tcmalloc::pageheap_free against systemstatistics-process_memory::maxrss? If that is the case, then there is no issue with fragmentation, because even though the address space is fragmented, the "holes" have mostly been returned to the OS for other processes to use ("unmapped").</div><div><br class=""></div><div>The point that Roland made also applies here: we are looking at the max across all processes and assuming that every process is the same. It's possible that one process has a high unmapped curve, but another has a high rss curve, and we don't see this on the plot. We would have to do 1D output of the grid arrays and plot each process separately to see the full detail. One way to see if this is necessary would be to plot both the max and min instead of just the max. That way, we can see if this is likely to be an issue.</div><div><br class=""></div></div><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">-- <br class="">Ian Hinder<br class=""><a href="https://ianhinder.net" class="">https://ianhinder.net</a><br class=""></div></div>
</div>
<br class=""></body></html>