hi Ian,
The memory seems to reach a steady state by iteration ~3000. Can you run an example where it dies with an OOM?
the OOM cases that i had were done in our local cluster (where i haven't compiled with tcmalloc); those were just higher resolution versions of this same simulation, where the OOM would be triggered around the time of one of these memory increases (ie, after iteration 2000 in this case, i'd guess).
From what I see here, the amount of memory allocated by Cactus grows then reaches a steady state (the green curve). This is not accounted for in the gridfunctions, so it must be from something else. I don't know what it might be. Does this run also have HDF5 output disabled?
yes, there was no HDF5 nor CarpetIOASCII output.
Can you check this by plotting tcmalloc::generic_current_allocated + tcmalloc::pageheap_free against systemstatistics-process_memory::maxrss? If that is the case, then there is no issue with fragmentation, because even though the address space is fragmented, the "holes" have mostly been returned to the OS for other processes to use ("unmapped").
sure, i've attached a plot with this.
The point that Roland made also applies here: we are looking at the max across all processes and assuming that every process is the same. It's possible that one process has a high unmapped curve, but another has a high rss curve, and we don't see this on the plot. We would have to do 1D output of the grid arrays and plot each process separately to see the full detail. One way to see if this is necessary would be to plot both the max and min instead of just the max. That way, we can see if this is likely to be an issue.
ok, i'm attaching another plot with both the min (dashed lines) and the max (full lines) plotted. i hope it helps.
thanks again, Miguel