<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 19 Feb 2013, at 19:19, Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Bernard<div><br></div><div>Did you set the parameters that Ian Hinder suggested? This will make interpreting the timing output much easier.</div><div><br></div><div style="">Time is not only spent in routines, but also in infrastructure tasks not listed here (regridding etc.). Ian Hinder implemented a tree-form timer output that makes it much easier to see how much time is spent on what task. I don't recall the options to select this -- Ian, is this Carpet::output_timer_tree_every and friends?</div></div></blockquote><div><br></div><div>Yes. That one displays the timer tree of the current process to standard output, and is useful for a quick overview. I recommend enabling it in all simulations. Unfortunately, it doesn't do a reduction across processes. You can also output the top N timers, not in tree form, using n_top_timers; these *are* reduced across processes, but because many of the timers overlap (e.g. some timers contain other timers), it is harder to get a good overview. You can also output an XML file per process containing the full hierarchical timer tree, which you can process in some other tool to do the reduction across processes (I have code in Mathematica to do this, if you are interested).</div><div><br></div><div>Ideally, I would like the stdout timer tree to include reductions across processes (min/max/average), and to have a version of the XML output which was also similarly reduced. The reason this was not easy to do was that you could have different timers existing on each process, so some reductions wouldn't work. It might be that this is no longer the case, and that implementing the reductions would be more straightforward now. Erik?</div><br><blockquote type="cite"><div dir="ltr">
<div><br></div><div style="">-erik</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 18, 2013 at 2:59 PM, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] <span dir="ltr"><<a href="mailto:bernard.j.kelly@nasa.gov" target="_blank">bernard.j.kelly@nasa.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">Hi Roland, and thanks for your reply. I'm still a bit confused, I confess<br>
(see below) ...<br>
<br>
On 2/16/13 12:38 AM, "Roland Haas" <<a href="mailto:roland.haas@physics.gatech.edu">roland.haas@physics.gatech.edu</a>> wrote:<br>
<br>
</div><div class="im">>Hello Bernard,<br>
><br>
>> Hi. Can someone explain to me why ML_BSSN calls SelectBoundConds within<br>
>> MoL_PostStep? It seems like the kind of once-off routine that would<br>
>>happen<br>
>> near the start of a simulation, rather than something that has to be<br>
>> performed every single timestep.<br>
>In the "new" boundary/symmetry interface (ie using thorn Boundary and<br>
>Symbase) one has to Select the variables for boundaries each time before<br>
>ApplyBCs is scheduled. There is a routine in Boundaries that clears the<br>
>selection.<br>
<br>
</div><div class="im">OK; but can you tell me *why* this is? Why should we ever have to<br>
re-specify what kind of boundary conditions we use during a simulation,<br>
any more than we re-specify the evolution equations? Perhaps I don't<br>
really understand what "select the variables" means here.<br>
<br>
><br>
</div><div class="im">>> I wouldn't mind, but while trying to understand why ML_BSSN was evolving<br>
>> so slowly on one of our machines, I looked at the TimerReport files, and<br>
>> saw that SelectBoundConds was taking *much* more time (like 20 times as<br>
>> long) than the actual RHS calculation routines.<br>
>The long time is most likely caused by the fact that the boundary<br>
>selection routine tends to be the one calling SYNC which means it is the<br>
>one that does an MPI wait (if there is load imbalance) and communicates<br>
>data for buffer zone prolongation etc.<br>
<br>
</div><div class="im">So it might be spending most of the time waiting for other cores to catch<br>
</div>up? But if it's really waiting for prior routines to finish on other<br>
<div class="im">processors, then on the handful of cores where SBC appears significantly<br>
*quicker* than usual (e.g. ~50,000 seconds instead of ~100,000) I should<br>
see earlier routines taking correspondingly *longer*, right? But I don't.<br>
<br>
</div><div class="im">I'm attaching TimerReport files for two cores on the same (128-core)<br>
</div>evolution. Core 000 is typical. Line 3591 (the most up-to-date instance of<br>
<div class="HOEnZb"><div class="h5">"large" SBC behaviour) shows about 100K seconds spent cumulatively over<br>
the simulation so far. Core 052 shows only about half as much time used in<br>
the same routine, but I can't see what other EVOL routines might be taking<br>
up the slack.<br>
<br>
(Note, BTW, that what I'm running isn't vanilla ML_BSSN, but a locally<br>
modified version called MH_BSSN. The scheduling and most routines are<br>
almost identical to McLachlan)<br>
<br>
Bernard<br>
<br>
><br>
>Yours,<br>
>Roland<br>
<br>
</div></div><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a>
</div>
_______________________________________________<br>Users mailing list<br><a href="mailto:Users@einsteintoolkit.org">Users@einsteintoolkit.org</a><br>http://lists.einsteintoolkit.org/mailman/listinfo/users<br></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- </div><div>Ian Hinder</div><div><a href="http://numrel.aei.mpg.de/people/hinder">http://numrel.aei.mpg.de/people/hinder</a></div></div></span></div></span></div></span></span>
</div>
<br></body></html>