<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 8 Aug 2018, at 11:41, Miguel Zilhão <<a href="mailto:miguel.zilhao.nogueira@tecnico.ulisboa.pt" class="">miguel.zilhao.nogueira@tecnico.ulisboa.pt</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">hi Ian,<br class=""><br class=""><blockquote type="cite" class="">The memory seems to reach a steady state by iteration ~3000. Can you run an example where it dies with an OOM?<br class=""></blockquote><br class="">the OOM cases that i had were done in our local cluster (where i haven't compiled with tcmalloc); those were just higher resolution versions of this same simulation, where the OOM would be triggered around the time of one of these memory increases (ie, after iteration 2000 in this case, i'd guess).<br class=""></div></div></blockquote><div><br class=""></div><div>Hi Miguel,</div><div><br class=""></div><div>The memory problems are very likely strongly related to the machine you run on. I don't know that we can take much information from a smaller test run on a different machine. We already see from this run that Carpet is not "leaking" memory continuously; the curves for allocated memory show what has been malloced and not freed, and it remains more or less constant after the initial phase.</div><div><br class=""></div><div>I think it's worth trying to get tcmalloc running on the cluster. So this means that you have never seen the OOM happen when using tcmalloc. It's possible that the improved memory allocation in tcmalloc over glibc would entirely solve the problem. </div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class="">Can you check this by plotting tcmalloc::generic_current_allocated + tcmalloc::pageheap_free against systemstatistics-process_memory::maxrss? If that is the case, then there is no issue with fragmentation, because even though the address space is fragmented, the "holes" have mostly been returned to the OS for other processes to use ("unmapped").<br class=""></blockquote><br class="">sure, i've attached a plot with this.<br class=""></div></div></blockquote><div><br class=""></div><div>Sorry, I made a mistake. It should have been pageheap_unmapped, not pageheap_free. Sorry! pageheap_free is essentially zero, and cannot account for the difference.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class="">The point that Roland made also applies here: we are looking at the max across all processes and assuming that every process is the same. It's possible that one process has a high unmapped curve, but another has a high rss curve, and we don't see this on the plot. We would have to do 1D output of the grid arrays and plot each process separately to see the full detail. One way to see if this is necessary would be to plot both the max and min instead of just the max. That way, we can see if this is likely to be an issue.<br class=""></blockquote><br class="">ok, i'm attaching another plot with both the min (dashed lines) and the max (full lines) plotted. i hope it helps.<br class=""></div></div></blockquote><div><br class=""></div><div>Thanks. This shows that the gridfunction usage is more or less similar across all processes, which is good. However, there is significant variation in most of the other quantities across processes. To understand this better, we would have to look at 1D ASCII output of the grid arrays, which is a bit painful to plot in gnuplot. Before this, I would definitely try to get tcmalloc running and outputting this information on the cluster in a run that actually shows the OOM. My guess is that you won't get an OOM with tcmalloc, and all will be fine :)</div></div><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">-- <br class="">Ian Hinder<br class=""><a href="https://ianhinder.net" class="">https://ianhinder.net</a><br class=""></div></div>
</div>
<br class=""></body></html>