<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>
<div>Hi Ian. Thanks for your response. To answer some of your questions:</div>
<div><br>
</div>
<div>* the behaviour is global --- it's affecting *every* core on *every* node, so it's not just a matter of uneven allocation of resources. </div>
<div><br>
</div>
<div>* I'm trying to get OpenMP to work on this machine, but my initial tests with it showed only modest improvements in memory efficiency (and didn't prevent this global memory increase)</div>
<div><br>
</div>
<div>* These aren't really very small memory effects --- the 1.5% and 3.0% are percentages of the *total* node memory. I was originally doing a vacuum + matter run, involving ~ twice as much memory to begin with. The doubling after first regridding then brought
me to something close to 100% of the nominal available memory on the node, and it was enough to kill the run. I removed the matter components to see if they were misbehaving, but they're not (at least not any more than the vacuum). Obviously, I can just use
more nodes, but I'm trying to understand the problem.</div>
<div><br>
</div>
<div>I'll try out SystemStatistics to see what it tells me.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Bernard</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ian Hinder <<a href="mailto:ian.hinder@aei.mpg.de">ian.hinder@aei.mpg.de</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, September 11, 2013 4:23 AM<br>
<span style="font-weight:bold">To: </span>Bernard Kelly <<a href="mailto:bernard.j.kelly@nasa.gov">bernard.j.kelly@nasa.gov</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:users@einsteintoolkit.org">users@einsteintoolkit.org</a>" <<a href="mailto:users@einsteintoolkit.org">users@einsteintoolkit.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Users] reported vs real memory usage + CarpetRegrid2?<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<br>
<div>
<div>On 10 Sep 2013, at 18:34, "Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]" <<a href="mailto:bernard.j.kelly@nasa.gov">bernard.j.kelly@nasa.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">Hi.<br>
<br>
I'm running a vacuum BHB evolution with a larger-than-usual set of inner<br>
refinement regions (levels 8, 9, 10 11 have radii of 12M, 8M, 6M, and 4M,<br>
respectively) and consequently the memory usage is a bit higher than<br>
normal. But I'm finding that it jumps up almost 100% after the first<br>
regridding, and stays there.<br>
<br>
My diagnostic for this is the result of top on each of the nodes (via<br>
"qtop.pl", a script on the machine I'm using). Sampled before the first<br>
regridding, it shows each core using ~ 1.5% of the node's total memory,<br>
while after regridding, it's more like 2.9% (these are Sandy Bridge nodes,<br>
with 16 available cores).<br>
<br>
However, the periodic output message from Carpet reporting the Grid<br>
structure etc. shows regions only marginally larger than before, and ---<br>
crucially for me --- has a marginally larger "Total required memory" (164<br>
GB -> 167 GB, for instance).<br>
<br>
So (a) what's using the extra memory, and (b) why isn't Carpet reporting<br>
it? How seriously should I be taking that "Total required memory" message?<br>
<br>
I see this with executables generated from both the last (ET_2012_11) and<br>
current (ET_2013_05) stable releases, BTW. I'm attaching the current<br>
parameter file and SCROUT from a run.<br>
</blockquote>
<div><br>
</div>
<div>I know there was a problem related to drastically increased memory usage, but I thought that this was introduced to the trunk after ET_2013_05, and that Erik had already fixed it. There was another problem in (I believe) ET_2012_11 where Carpet was always
collapsing multiple grids on a refinement level into the smallest enclosing box, leading to huge memory usage, but that was also fixed, and I believe backported to ET_2012_11. Have you tried using the SystemStatistics thorn to monitor memory usage? This
should be easier than using top on the nodes.</div>
<div><br>
</div>
<div>How is the grid distributed among the nodes? Even if the total required memory is roughly constant, it's possible that the grid is distributed unevenly between nodes. Is every node showing the increased memory usage? Given that you are still using a
very small amount of memory, it's possible that Carpet is just "overallocating" on the first regridding as it anticipates that it might need more memory later, and memory allocation might be expensive. The amount of overallocation is presumably small in comparison
to the total available memory, but might be of the order of 1%, as you are seeing.</div>
<div><br>
</div>
<div>I wouldn't worry about this small amount of increased memory usage unless you can reproduce the problem on a more heavily-loaded system. From your description, I suspect you are not using OpenMP. Why is that? Using pure MPI leads to an unnecessary memory
overhead.</div>
<div><br>
</div>
</div>
<div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; border-spacing: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>-- </div>
<div>Ian Hinder</div>
<div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div>
</div>
</span></div>
</span></div>
</span></span></div>
<br>
</div>
</div>
</span>
</body>
</html>