[Einstein Toolkit] #1412: Memory leak on development branch

List overview All Threads
Download

newer

older

[Einstein Toolkit] #1372:...

[Einstein Toolkit] #1445: add tags...

Einstein Toolkit

2 Aug 2013 2 Aug '13

5:37 a.m.

#1412: Memory leak on development branch -----------------------------------+---------------------------------------- Reporter: hopper.seth@… | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: development version Keywords: | -----------------------------------+---------------------------------------- I am seeing what appears to be a memory leak that has showed up since the Gauss release. I have run the same parameter file on Gauss and the development branch, and see linear growth in memory usage in the development version. From standard out it looks to be a bug in Carpet, as the memory usage jumps after re-gridding.

I've attached:

- The parameter file, bbhCart2-3.rpar. It runs on 12 cores on Datura. - Plots showing the memory usage per process in the two cases. - Standard out from the two cases.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

2 Aug 2 Aug

5:48 a.m.

#1412: Memory leak on development branch ------------------------------------+--------------------------------------- Reporter: hopper.seth@… | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: development version Resolution: | Keywords: ------------------------------------+---------------------------------------

Comment (by hopper.seth@…):

Standard out was too big to upload. Let me know if you want it, and I can send it along.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5:59 a.m.

Comment (by hopper.seth@…):

Okay, I zipped the standard out files and now I've been able to upload them. -Seth

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10:37 a.m.

Comment (by eschnett):

Please attach the actual parameter file, just to be sure.

Please attach your thorn list and option list as well, as well as any Simfactory options you used while configuring and building, and which may override some settings. Maybe sending the captured OptionList and ThornList would be best.

Please attach the content of Cactus's config-data/make.config.defn, just to be safe.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

10:47 a.m.

Comment (by eschnett):

Can you also attach the output describing how much memory is used? This is probably the source you used for the pdfs.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:11 a.m.

Comment (by hopper.seth@…):

Okay, I think I've uploaded everything you asked for. Let me know if you need anything I missed. Thanks for looking into this Erik! - Seth

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:39 p.m.

Comment (by eschnett):

I see that you are not using the new bboxset2 class, so this new algorithm is not responsible for this memory leak.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4 Aug 4 Aug

10:52 a.m.

Comment (by eschnett):

I can reproduce the problem, even on a single MPI process and with a single OpenMP thread. It does not seem to occur with gcc, only with Intel.

I corrected a Carpet (or Intel?) problem where memory may have become corrupted, but this does not make the problem go away. The amount of memory actually used (and reachable by) Carpet stays approximately constant.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6 Aug 6 Aug

11:46 a.m.

Comment (by rhaas):

This seems to be caused by either SYNC or calls to boundary. The parfile vacuum5.par shows linear growth in memory usage, the parfile nothing.par does not. The difference is including ADMBase and "static" evolution in the parfile or not.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

11:52 a.m.

Comment (by eschnett):

Yay! Thanks for paring this down.

Which compiler/machine was this? I'm not completely sure yet that this effect is generic.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:22 p.m.

Comment (by rhaas):

Zwicky, intel 11 (icpc --version 11.1 20100414), option list from simfactory (run is in /panfs/ds06/sxs/rhaas/cactus/runs/vacuum5). Run was using 2 processes, 6 threads each. There is beautiful linear growth in memory consumption from step 0.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:10 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:27 p.m.

Comment (by hinder):

Seth's runs were all using Intel as well; the current simfactory configurations for datura.cfg and supermuc.cfg. Note that the problem is not present in the Gauss release; if reading the commits since then doesn't throw up something obvious, you could also try bisecting. See http://git.barrywardell.net/simfactory2.git/blob/HEAD:/bin/bisect-test.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:11 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:44 p.m.

Comment (by rhaas):

Given the signature of the memory leak it might actually be easier to run something like mtrace (http://en.wikipedia.org/wiki/Mtrace) and then look for places in the code that allocate lots of memory (identifyable in the matrace output) since in fact no memory might be leaked if "all" that happens is the a C++ container grows without bounds since the container would free itself before program end.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:12 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4:09 p.m.

Comment (by eschnett):

... and here we go -- the latest results! Looking at SystemStatistics (thanks, Ian!), in particular at {{{ int uordblks This is the total size of memory occupied by chunks handed out by malloc. int fordblks This is the total size of memory occupied by free (not in use) chunks. }}} uordblks is remaining constant at 800 MB, fordblks is increasing linearly in time at a rate of about 100 MB / 256 iterations.

This means that malloc is not reusing memory, and thus the overhead of libc increases with time. We encountered this before; the communication pattern of CarpetIOASCII triggered this. We introduced "memory pools" where we allocated large chunks of memory ahead of time, serving this when allocating the communication buffers, and freeing them when CarpetIOASCII is finished. This works fine.

I assume we need to do the same for some other thorn. Don't know yet where.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:13 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

7 Aug 7 Aug

9:31 p.m.

Comment (by eschnett):

More notes:

It is not a memory leak; only libc's free memory increases.

The memory increases in jumps of a few MByte. These jumps occur near mode switches or near synchronization.

The problem only occurs with Intel, not with gcc. With Intel, the libc free memory increases linearly, up to 1.4 GByte in one particular test case. With gcc, the libc free memory stays bounded at about 29 MByte.

It is not due to RotatingSymmetry180. It is not due to the routine SyncGroups. It is not due to Carpet's timers.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:14 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 Aug 8 Aug

1:21 a.m.

Comment (by rhaas):

In case it helps: I also see a linear increase in the memory consumption (maxrss_mb) when using gcc 4.7 on my workstation for the vacuum file. So far memory consumption has increased from ~450MB to ~7GB, flattening off a little bit at the end but certainly still increasing.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:15 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:30 a.m.

Comment (by eschnett):

The culprit seems to be ggf::ref_bnd_prolongate_all or one of its children.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:16 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:19 p.m.

Comment (by eschnett):

The problem was triggered by LoopControl. Allocating new lc_setup_t or lc_params_t caused libc's "free" memory to increase. I am now using a memory pool there, which circumvents the problem.

This is new corrected in the development version, which also introduces the new Timer thorn.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:17 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

30 Sep 30 Sep

12:26 a.m.

#1412: Memory leak on development branch ------------------------------------+--------------------------------------- Reporter: hopper.seth@… | Owner: eschnett Type: defect | Status: closed Priority: major | Milestone: Component: Carpet | Version: development version Resolution: fixed | Keywords: ------------------------------------+--------------------------------------- Changes (by rhaas):

* status: new => closed * resolution: => fixed

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1412#comment:18 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

4757

Age (days ago)

4816

Last active (days ago)

trac@lists.einsteintoolkit.org

18 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit