<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; "><div><div>Hi Ian.</div><div><div><br></div><div>The TimerReport.XXXXXX.txt files have no instance of "Barrier" (any capitalisation) appearing. Does it depend on anything else apart from the parameters you've specified? I'm attaching one process's TimerReport (only the last time output, for compactness), and the associated parfile here.</div><div><br></div><div>Bernard</div><div><div></div></div></div></div><div><br></div><span id="OLK_SRC_BODY_SECTION"><div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"><span style="font-weight:bold">From: </span> Ian Hinder <<a href="mailto:ian.hinder@aei.mpg.de">ian.hinder@aei.mpg.de</a>><br><span style="font-weight:bold">Date: </span> Tuesday, February 19, 2013 4:16 PM<br><span style="font-weight:bold">To: </span> Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>><br><span style="font-weight:bold">Cc: </span> Bernard Kelly <<a href="mailto:bernard.j.kelly@nasa.gov">bernard.j.kelly@nasa.gov</a>>, "<a href="mailto:users@einsteintoolkit.org">users@einsteintoolkit.org</a>" <<a href="mailto:users@einsteintoolkit.org">users@einsteintoolkit.org</a>><br><span style="font-weight:bold">Subject: </span> Re: [Users] logic of scheduling SelectBoundConds in McLachlan?<br></div><div><br></div><div><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 19 Feb 2013, at 20:16, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">On Tue, Feb 19, 2013 at 1:24 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] <span dir="ltr"><<a href="mailto:bernard.j.kelly@nasa.gov" target="_blank">bernard.j.kelly@nasa.gov</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Ian (and Frank and Erik). Thanks for the further insight on the<br>
profiling.<br><br><br>
[Please ignore the new mail that just came through with the 400KB<br>
attachment. That was my first attempt that was held for moderation because<br>
of the attachment size. Then I sent the slimmed-down attachments, but this<br>
was still in the pipeline.]<br><br>
I was looking at *all* the processor outputs (that is, all the<br>
TimerReport_XXXXXX files), but not necessarily at all fields in all of<br>
them. I concentrated on the CCTK_EVOL section of the report, and then only<br>
looked closely at discrepancies between a sample "longer SelectBoundCond"<br>
processor and each of the five or six "shorter SelectBoundcond"<br>
processors. I suppose to do a more complete job, I'd have to start<br>
scripting ...<br><br>
Anyway, I *hadn't* been using those profiling parameters before, so my<br>
conclusions were probably dodgy as you say. After your reply I re-enabled<br>
them and restarted the run. Since it's so slow, I'm now looking at the<br>
TimerReports from earlier in the new run, and no longer see any<br>
discrepancies between different processors (that is, there don't seem to<br>
be any "shorter SelectBoundcond" processors any more).<br><br>
So if *all* the processors are showing essentially the same information,<br>
and the "schedule_barriers" and "sync_barriers" are in place, then there's<br>
no significant load imbalance? And yet it is slow as hell ...<br></blockquote><div><br></div><div style="">With schedule barriers, load imbalance is hidden in these barriers. That is, you would need to measure how much time each process spends in these barriers. I expect that some processes will spend 0s there, while others will spend 50,000s there. That would be your load imbalance.</div></div></div></div></blockquote><div><br></div><div>When I added the sync barriers, I added timers on all the barriers. You should see timer entries named ".../Barrier". Do you see these, and are they taking a lot of time? The timer names are hierarchical, so you should be able to see which function barriers are causing the slowdown.</div><div><br></div><div>When I have done tests using schedule barriers, they did not impose a huge penalty like the one you are describing. Maybe 30%, no more. </div></div><br><div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; border-spacing: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span></div><br></div></div></span></body></html>