<div dir="ltr">Yosef, James<div><br></div><div>I'd like to at least try to see whether the McLachlan performance you're seeing is in the right ballpark. Can you post the parameter file you're using?</div><div><br></div><div>-erik</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 28, 2015 at 9:19 AM, Yosef Zlochower <span dir="ltr"><<a href="mailto:yosef@astro.rit.edu" target="_blank">yosef@astro.rit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Erik,<br>
<br>
Our standard choice is to use 8th order centered<br>
differencing with 5th-order dissipation. For this<br>
test, Jim used 8th-order with upwinding and<br>
thus used 5 ghost zones (rather than 4).<br>
We use the full complement of 4*ghost buffer<br>
zones. Near the boundary we reduce the order of the<br>
stencil from 8 to 6 to 4 to 2 and on the boundary<br>
we apply Sommerfeld boundary conditions (implemented as<br>
modifications to RHS).<br>
<br>
For the gauge, we evolve<br>
d0 alpha = -2 alpha K<br>
and<br>
dt beta = 3/4 Gamma^i<br>
(we don't introduce the auxiliary B field).<br>
<br>
We precalculate the mixed derivatives and<br>
only calculate first derivatives and non-mixed second<br>
derivatives in the main loop. We apply<br>
dissipation to all variables, but apply it<br>
one variable at a time (i.e., we go through<br>
a full 3d loop nvars times).<br>
<br>
<br>
The conformal factor is W = sqrt(chi).<br>
<br>
We use the admbase variables alpha and beta<br>
and fill in kij and gij after each ministep<br>
of the RK4 integrator. We similarly enforce<br>
the algebraic constraints at each ministep.<br>
We calculate the constraints using the BSSN<br>
variables, but use a 4th order stencil and implement<br>
mixed derivatives using one dimensional derivatives<br>
along the diagonal (this is less accurate, but<br>
faster).<br>
<br>
<br>
The FD order is a compile time option and<br>
we can test the code with and without upwinding,<br>
with FD orders of 2nd through 8, and we can<br>
use the standard or "quick" mixed derivatives<br>
for the constraint. We can similarly use<br>
3, 5, 7, 9 order dissipation.<br>
<br>
We use openmp parallelization along the "z" direction<br>
of our loops<span class=""><br>
<br>
<br>
<br>
<br>
On 07/27/2015 10:21 PM, Erik Schnetter wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jim<br>
<br><span class="">
By default, McLachlan is configured to be generic, i.e. many choices can<br>
be made at run time. For example, there are 4 additional gauge variables<br>
evolved (A and B^i). If you are willing to restrict your run-time<br>
choices, for example by fixing derivative order, disabling multi-block<br>
systems, choosing a particular gauge, choice of conformal factor, etc.,<br>
then this will lead to a more efficient code. You can easily do so for<br>
McLachlan by generating a new version in Kranc. If you look in the<br>
beginning of McLachlan_BSSN.m in the section "Choose code to generate",<br>
then you'll see the parameters you can choose. We have made a particular<br>
set of choices for a configuration "BSSN_bench", but these are<br>
apparently different from LazEv. For example, we use 4th order finite<br>
differencing, and evolve B^i in time as well.<br>
<br>
Can you specify all the details of your formulation? I notice that you<br>
didn't respond to my request to point me to the source code, and there<br>
are quite a few details that you also didn't describe yet (and are also<br>
not described in your paper), such as your choice of conformal factor,<br>
whether/how your code is integrated with ADMBase, whether the<br>
constraints are calculated from the BSSN or the ADM variables, how<br>
advection derivatives are handled, etc. For a true comparison, this all<br>
needs to be known. Unless we both know exactly what both the codes do,<br>
we'll never know where the physics differences are.<br>
<br>
-erik<br>
<br>
<br>
On Mon, Jul 27, 2015 at 5:39 PM, James Healy <<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a><br></span><div><div class="h5">
<mailto:<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a>>> wrote:<br>
<br>
Hi Erik,<br>
<br>
We evolve alpha and beta for the gauge variables as given in<br>
equations 7a and 7b of <a href="http://arxiv.org/pdf/1506.06153.pdf" rel="noreferrer" target="_blank">http://arxiv.org/pdf/1506.06153.pdf</a>. The<br>
number I reported for LazEv was using 5th order dissipation applied<br>
to all 21 evolved variables (alp, beta^i, At_{ij}, gt_{ij},<br>
Gammat^{i}, trK, and xi).<br>
<br>
LazEv is calculating the Hamiltonian and Momentum constraints as<br>
well as the constraints on Gammat^i using 4th order stencils.<br>
<br>
I also forgot to mention that I did these tests on 16 nodes, and the<br>
memory usage was relatively low ( ~2 GB/MPI process ).<br>
<br>
I will give the internal dissipation for McLachlan a try.<br>
<br>
Thanks,<br>
Jim<br>
<br>
<br>
On 07/27/2015 05:56 PM, Erik Schnetter wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Jim<br>
<br>
Thanks for posting the details.<br>
<br>
Can you give us more details about the LazEv scheme? In particular<br>
there may be differences in the gauge. What (gauge) variables do<br>
you evolve? What gauge conditions do you use? And what kind of<br>
dissipation do you apply? Can you point us to the source code?<br>
<br>
For the new McLachlan, you would probably use the built-in<br>
dissipation instead of thorn Dissipation, which should lead to a<br>
small speed-up.<br>
<br>
-erik<br>
<br>
On Mon, Jul 27, 2015 at 4:43 PM, James Healy <<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a><br></div></div><div><div class="h5">
<mailto:<a href="mailto:jchsma@rit.edu" target="_blank">jchsma@rit.edu</a>>> wrote:<br>
<br>
Hello,<br>
<br>
I have been running some tests on Stampede comparing the run<br>
speed of McLachlan to RIT's evolution thorn LazEv. I started<br>
with the qc0-mclachlan.par parameter file included with the<br>
Einstein Toolkit, added a few refinement levels, increased the<br>
resolution and changed McLachlan to be 8th order (and<br>
increased the number of ghost zones to 5). I also increased<br>
the initial separation so the finest grids aren't already<br>
overlapping. To compare with LazEv, I removed the McLachlan<br>
and Dissipation thorns and replaced them with LazEv.<br>
Everything else in the parameter file is exactly the same. I<br>
tried using both the McLachlan master and rewrite branches.<br>
<br>
The grid setup is 10 levels of refinement, dx=4M on the<br>
coarsest with outer boundary at 400M, M/128 on the finest with<br>
r=0.6M, CFL is 0.25. Both use 8th order spatial differencing<br>
with ghost_size=5 and 5th order dissipation.<br>
<br>
Below is a summary of the results as reported at iteration 256<br>
from Carpet::physical_time_per_hour:<br>
<br>
McLachlan - rewrite branch: 3.0596110 M/hr<br>
McLachlan - master branch: 3.8033607 M/hr<br>
LazEv - 4.1941544 M/hr<br>
<br>
I am using the stampede-impi.cfg configuration file in<br>
simfactory. "module list" returns:<br>
<br>
1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7) TACC<br></div></div>
2) Linux 4) intel/<a href="http://13.0.2.146" rel="noreferrer" target="_blank">13.0.2.146</a> <<a href="http://13.0.2.146" rel="noreferrer" target="_blank">http://13.0.2.146</a>> 6)<br>
cluster 8) impi/<a href="http://4.1.0.030" rel="noreferrer" target="_blank">4.1.0.030</a> <<a href="http://4.1.0.030" rel="noreferrer" target="_blank">http://4.1.0.030</a>><div><div class="h5"><br>
<br>
Attached is my parameter file. I pasted the McLachlan<br>
parameters below. Are there any optimizations that I can use<br>
for McLachlan? Are the parameters I am using for it what would<br>
be used for production runs?<br>
<br>
ML_BSSN::harmonicN = 1 # 1+log<br>
ML_BSSN::harmonicF = 2.0 # 1+log<br>
ML_BSSN::ShiftGammaCoeff = 0.75<br>
ML_BSSN::BetaDriver = 1.0<br>
ML_BSSN::LapseAdvectionCoeff = 1.0<br>
ML_BSSN::ShiftAdvectionCoeff = 1.0<br>
<br>
ML_BSSN::MinimumLapse = 1.0e-8<br>
<br>
ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"<br>
ML_BSSN::my_rhs_boundary_condition = "NewRad"<br>
Boundary::radpower = 2<br>
<br>
ML_BSSN::ML_log_confac_bound = "none"<br>
ML_BSSN::ML_metric_bound = "none"<br>
ML_BSSN::ML_Gamma_bound = "none"<br>
ML_BSSN::ML_trace_curv_bound = "none"<br>
ML_BSSN::ML_curv_bound = "none"<br>
ML_BSSN::ML_lapse_bound = "none"<br>
ML_BSSN::ML_dtlapse_bound = "none"<br>
ML_BSSN::ML_shift_bound = "none"<br>
ML_BSSN::ML_dtshift_bound = "none"<br>
<br>
ML_BSSN::fdOrder = 8<br>
<br>
ActiveThorns = "Dissipation"<br>
<br>
Dissipation::order = 5<br>
Dissipation::vars = "<br>
ML_BSSN::ML_metric<br>
ML_BSSN::ML_trace_curv<br>
ML_BSSN::ML_curv<br>
ML_BSSN::ML_Gamma<br>
ML_BSSN::ML_lapse<br>
ML_BSSN::ML_shift<br>
ML_BSSN::ML_dtlapse<br>
ML_BSSN::ML_dtshift<br>
"<br>
<br>
ActiveThorns = "ML_ADMConstraints"<br>
<br>
<br>
Thanks,<br>
Jim Healy<br>
<br>
_______________________________________________<br>
Users mailing list<br></div></div>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a> <mailto:<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a>><span class=""><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
<br>
<br>
<br>
--<br></span>
Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a> <mailto:<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" rel="noreferrer" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br>
</blockquote>
<br>
<br>
<br>
<br>
--<br>
Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a> <mailto:<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" rel="noreferrer" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br>
<br>
<br><span class="">
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
</span></blockquote>
<br>
<br>
-- <br>
Dr. Yosef Zlochower<br>
Center for Computational Relativity and Gravitation<br>
Associate Professor<br>
School of Mathematical Sciences<br>
Rochester Institute of Technology<br>
85 Lomb Memorial Drive<br>
Rochester, NY 14623<br>
<br>
Office:74-2067<br>
Phone: <a href="tel:%2B1%20585-475-6103" value="+15854756103" target="_blank">+1 585-475-6103</a><br>
<br>
<a href="mailto:yosef@astro.rit.edu" target="_blank">yosef@astro.rit.edu</a><br>
<br>
CONFIDENTIALITY NOTE: The information transmitted, including<br>
attachments, is intended only for the person(s) or entity to which it<br>
is addressed and may contain confidential and/or privileged material.<br>
Any review, retransmission, dissemination or other use of, or taking<br>
of any action in reliance upon this information by persons or entities<br>
other than the intended recipient is prohibited. If you received this<br>
in error, please contact the sender and destroy any copies of this<br>
information.<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br><a href="http://www.perimeterinstitute.ca/personal/eschnetter/" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a></div>
</div>