<div dir="ltr">Jim<div><br></div><div>By default, McLachlan is configured to be generic, i.e. many choices can be made at run time. For example, there are 4 additional gauge variables evolved (A and B^i). If you are willing to restrict your run-time choices, for example by fixing derivative order, disabling multi-block systems, choosing a particular gauge, choice of conformal factor, etc., then this will lead to a more efficient code. You can easily do so for McLachlan by generating a new version in Kranc. If you look in the beginning of McLachlan_BSSN.m in the section "Choose code to generate", then you'll see the parameters you can choose. We have made a particular set of choices for a configuration "BSSN_bench", but these are apparently different from LazEv. For example, we use 4th order finite differencing, and evolve B^i in time as well.</div><div><br></div><div>Can you specify all the details of your formulation? I notice that you didn't respond to my request to point me to the source code, and there are quite a few details that you also didn't describe yet (and are also not described in your paper), such as your choice of conformal factor, whether/how your code is integrated with ADMBase, whether the constraints are calculated from the BSSN or the ADM variables, how advection derivatives are handled, etc. For a true comparison, this all needs to be known. Unless we both know exactly what both the codes do, we'll never know where the physics differences are.</div><div><br></div><div>-erik</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 27, 2015 at 5:39 PM, James Healy <span dir="ltr"><<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi Erik,<br>
<br>
We evolve alpha and beta for the gauge variables as given in
equations 7a and 7b of <a href="http://arxiv.org/pdf/1506.06153.pdf" target="_blank">http://arxiv.org/pdf/1506.06153.pdf</a>. The
number I reported for LazEv was using 5th order dissipation
applied to all 21 evolved variables (alp, beta^i, At_{ij},
gt_{ij}, Gammat^{i}, trK, and xi). <br>
<br>
LazEv is calculating the Hamiltonian and Momentum constraints as
well as the constraints on Gammat^i using 4th order stencils. <br>
<br>
I also forgot to mention that I did these tests on 16 nodes, and
the memory usage was relatively low ( ~2 GB/MPI process ).<br>
<br>
I will give the internal dissipation for McLachlan a try.<br>
<br>
Thanks,<br>
Jim<div><div class="h5"><br>
<br>
On 07/27/2015 05:56 PM, Erik Schnetter wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">Jim
<div><br>
</div>
<div>Thanks for posting the details.</div>
<div><br>
</div>
<div>Can you give us more details about the LazEv scheme? In
particular there may be differences in the gauge. What (gauge)
variables do you evolve? What gauge conditions do you use? And
what kind of dissipation do you apply? Can you point us to the
source code?</div>
<div><br>
</div>
<div>For the new McLachlan, you would probably use the built-in
dissipation instead of thorn Dissipation, which should lead to
a small speed-up.</div>
<div><br>
</div>
<div>-erik</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jul 27, 2015 at 4:43 PM, James
Healy <span dir="ltr"><<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I have been running some tests on Stampede comparing the run
speed of McLachlan to RIT's evolution thorn LazEv. I
started with the qc0-mclachlan.par parameter file included
with the Einstein Toolkit, added a few refinement levels,
increased the resolution and changed McLachlan to be 8th
order (and increased the number of ghost zones to 5). I
also increased the initial separation so the finest grids
aren't already overlapping. To compare with LazEv, I
removed the McLachlan and Dissipation thorns and replaced
them with LazEv. Everything else in the parameter file is
exactly the same. I tried using both the McLachlan master
and rewrite branches.<br>
<br>
The grid setup is 10 levels of refinement, dx=4M on the
coarsest with outer boundary at 400M, M/128 on the finest
with r=0.6M, CFL is 0.25. Both use 8th order spatial
differencing with ghost_size=5 and 5th order dissipation.<br>
<br>
Below is a summary of the results as reported at iteration
256 from Carpet::physical_time_per_hour:<br>
<br>
McLachlan - rewrite branch: 3.0596110 M/hr<br>
McLachlan - master branch: 3.8033607 M/hr<br>
LazEv - 4.1941544 M/hr<br>
<br>
I am using the stampede-impi.cfg configuration file in
simfactory. "module list" returns:<br>
<br>
1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7)
TACC<br>
2) Linux 4) intel/<a href="http://13.0.2.146" rel="noreferrer" target="_blank">13.0.2.146</a>
6) cluster 8) impi/<a href="http://4.1.0.030" rel="noreferrer" target="_blank">4.1.0.030</a><br>
<br>
Attached is my parameter file. I pasted the McLachlan
parameters below. Are there any optimizations that I can
use for McLachlan? Are the parameters I am using for it what
would be used for production runs?<br>
<br>
ML_BSSN::harmonicN = 1 # 1+log<br>
ML_BSSN::harmonicF = 2.0 # 1+log<br>
ML_BSSN::ShiftGammaCoeff = 0.75<br>
ML_BSSN::BetaDriver = 1.0<br>
ML_BSSN::LapseAdvectionCoeff = 1.0<br>
ML_BSSN::ShiftAdvectionCoeff = 1.0<br>
<br>
ML_BSSN::MinimumLapse = 1.0e-8<br>
<br>
ML_BSSN::my_initial_boundary_condition =
"extrapolate-gammas"<br>
ML_BSSN::my_rhs_boundary_condition = "NewRad"<br>
Boundary::radpower = 2<br>
<br>
ML_BSSN::ML_log_confac_bound = "none"<br>
ML_BSSN::ML_metric_bound = "none"<br>
ML_BSSN::ML_Gamma_bound = "none"<br>
ML_BSSN::ML_trace_curv_bound = "none"<br>
ML_BSSN::ML_curv_bound = "none"<br>
ML_BSSN::ML_lapse_bound = "none"<br>
ML_BSSN::ML_dtlapse_bound = "none"<br>
ML_BSSN::ML_shift_bound = "none"<br>
ML_BSSN::ML_dtshift_bound = "none"<br>
<br>
ML_BSSN::fdOrder = 8<br>
<br>
ActiveThorns = "Dissipation"<br>
<br>
Dissipation::order = 5<br>
Dissipation::vars = "<br>
ML_BSSN::ML_metric<br>
ML_BSSN::ML_trace_curv<br>
ML_BSSN::ML_curv<br>
ML_BSSN::ML_Gamma<br>
ML_BSSN::ML_lapse<br>
ML_BSSN::ML_shift<br>
ML_BSSN::ML_dtlapse<br>
ML_BSSN::ML_dtshift<br>
"<br>
<br>
ActiveThorns = "ML_ADMConstraints"<br>
<br>
<br>
Thanks,<br>
Jim Healy<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
</div>