<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 19 Feb 2013, at 20:16, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">On Tue, Feb 19, 2013 at 1:24 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] <span dir="ltr"><<a href="mailto:bernard.j.kelly@nasa.gov" target="_blank">bernard.j.kelly@nasa.gov</a>></span> wrote:<br>
<div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Ian (and Frank and Erik). Thanks for the further insight on the<br>
profiling.<br>
<br>
<br>
[Please ignore the new mail that just came through with the 400KB<br>
attachment. That was my first attempt that was held for moderation because<br>
of the attachment size. Then I sent the slimmed-down attachments, but this<br>
was still in the pipeline.]<br>
<br>
I was looking at *all* the processor outputs (that is, all the<br>
TimerReport_XXXXXX files), but not necessarily at all fields in all of<br>
them. I concentrated on the CCTK_EVOL section of the report, and then only<br>
looked closely at discrepancies between a sample "longer SelectBoundCond"<br>
processor and each of the five or six "shorter SelectBoundcond"<br>
processors. I suppose to do a more complete job, I'd have to start<br>
scripting ...<br>
<br>
Anyway, I *hadn't* been using those profiling parameters before, so my<br>
conclusions were probably dodgy as you say. After your reply I re-enabled<br>
them and restarted the run. Since it's so slow, I'm now looking at the<br>
TimerReports from earlier in the new run, and no longer see any<br>
discrepancies between different processors (that is, there don't seem to<br>
be any "shorter SelectBoundcond" processors any more).<br>
<br>
So if *all* the processors are showing essentially the same information,<br>
and the "schedule_barriers" and "sync_barriers" are in place, then there's<br>
no significant load imbalance? And yet it is slow as hell ...<br></blockquote><div><br></div><div style="">With schedule barriers, load imbalance is hidden in these barriers. That is, you would need to measure how much time each process spends in these barriers. I expect that some processes will spend 0s there, while others will spend 50,000s there. That would be your load imbalance.</div></div></div></div></blockquote><div><br></div><div>When I added the sync barriers, I added timers on all the barriers. You should see timer entries named ".../Barrier". Do you see these, and are they taking a lot of time? The timer names are hierarchical, so you should be able to see which function barriers are causing the slowdown.</div><div><br></div><div>When I have done tests using schedule barriers, they did not impose a huge penalty like the one you are describing. Maybe 30%, no more. </div></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></body></html>