[Users] logic of scheduling SelectBoundConds in McLachlan?

[Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]]

Hi Roland, and thanks for your reply. I'm still a bit confused, I confess
(see below) ...

On 2/16/13 12:38 AM, "Roland Haas" <roland.haas at physics.gatech.edu> wrote:

>Hello Bernard,
>
>> Hi. Can someone explain to me why ML_BSSN calls SelectBoundConds within
>> MoL_PostStep? It seems like the kind of once-off routine that would
>>happen
>> near the start of a simulation, rather than something that has to be
>> performed every single timestep.
>In the "new" boundary/symmetry interface (ie using thorn Boundary and
>Symbase) one has to Select the variables for boundaries each time before
>ApplyBCs is scheduled. There is a routine in Boundaries that clears the
>selection.

OK; but can you tell me *why* this is? Why should we ever have to
re-specify what kind of boundary conditions we use during a simulation,
any more than we re-specify the evolution equations? Perhaps I don't
really understand what "select the variables" means here.

>
>> I wouldn't mind, but while trying to understand why ML_BSSN was evolving
>> so slowly on one of our machines, I looked at the TimerReport files, and
>> saw that SelectBoundConds was taking *much* more time (like 20 times as
>> long) than the actual RHS calculation routines.
>The long time is most likely caused by the fact that the boundary
>selection routine tends to be the one calling SYNC which means it is the
>one that does an MPI wait (if there is load imbalance) and communicates
>data for buffer zone prolongation etc.

So it might be spending most of the time waiting for other cores to catch
up? But if it's really waiting for prior routines to finish on other
processors, then on the handful of cores where SBC appears significantly
*quicker* than usual (e.g. ~50,000 seconds instead of ~100,000) I should
see earlier routines taking correspondingly *longer*, right? But I don't.

I'm attaching TimerReport files for two cores on the same (128-core)
evolution. Core 000 is typical. Line 3591 (the most up-to-date instance of
"large" SBC behaviour) shows about 100K seconds spent cumulatively over
the simulation so far. Core 052 shows only about half as much time used in
the same routine, but I can't see what other EVOL routines might be taking
up the slack. 

(Note, BTW, that what I'm running isn't vanilla ML_BSSN, but a locally
modified version called MH_BSSN. The scheduling and most routines are
almost identical to McLachlan)

Bernard

>
>Yours,
>Roland

-------------- next part --------------
A non-text attachment was scrubbed...
Name: TimerReports_BJK.tgz
Type: application/octet-stream
Size: 338570 bytes
Desc: TimerReports_BJK.tgz
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20130218/3d9387ec/attachment-0001.obj