Hi Erik,
Our standard choice is to use 8th order centered
differencing with 5th-order dissipation. For this
test, Jim used 8th-order with upwinding and
thus used 5 ghost zones (rather than 4).
We use the full complement of 4*ghost buffer
zones. Near the boundary we reduce the order of the
stencil from 8 to 6 to 4 to 2 and on the boundary
we apply Sommerfeld boundary conditions (implemented as
modifications to RHS).
For the gauge, we evolve
d0 alpha = -2 alpha K
and
dt beta = 3/4 Gamma^i
(we don't introduce the auxiliary B field).
We precalculate the mixed derivatives and
only calculate first derivatives and non-mixed second
derivatives in the main loop. We apply
dissipation to all variables, but apply it
one variable at a time (i.e., we go through
a full 3d loop nvars times).
The conformal factor is W = sqrt(chi).
We use the admbase variables alpha and beta
and fill in kij and gij after each ministep
of the RK4 integrator. We similarly enforce
the algebraic constraints at each ministep.
We calculate the constraints using the BSSN
variables, but use a 4th order stencil and implement
mixed derivatives using one dimensional derivatives
along the diagonal (this is less accurate, but
faster).
The FD order is a compile time option and
we can test the code with and without upwinding,
with FD orders of 2nd through 8, and we can
use the standard or "quick" mixed derivatives
for the constraint. We can similarly use
3, 5, 7, 9 order dissipation.
We use openmp parallelization along the "z" direction
of our loops
On 07/27/2015 10:21 PM, Erik Schnetter wrote:
Jim
By default, McLachlan is configured to be generic, i.e. many choices can
be made at run time. For example, there are 4 additional gauge variables
evolved (A and B^i). If you are willing to restrict your run-time
choices, for example by fixing derivative order, disabling multi-block
systems, choosing a particular gauge, choice of conformal factor, etc.,
then this will lead to a more efficient code. You can easily do so for
McLachlan by generating a new version in Kranc. If you look in the
beginning of McLachlan_BSSN.m in the section "Choose code to generate",
then you'll see the parameters you can choose. We have made a particular
set of choices for a configuration "BSSN_bench", but these are
apparently different from LazEv. For example, we use 4th order finite
differencing, and evolve B^i in time as well.
Can you specify all the details of your formulation? I notice that you
didn't respond to my request to point me to the source code, and there
are quite a few details that you also didn't describe yet (and are also
not described in your paper), such as your choice of conformal factor,
whether/how your code is integrated with ADMBase, whether the
constraints are calculated from the BSSN or the ADM variables, how
advection derivatives are handled, etc. For a true comparison, this all
needs to be known. Unless we both know exactly what both the codes do,
we'll never know where the physics differences are.
-erik
On Mon, Jul 27, 2015 at 5:39 PM, James Healy <jchsma@rit.edu<mailto:jchsma@rit.edu>> wrote:
Hi Erik,
We evolve alpha and beta for the gauge variables as given in
equations 7a and 7b of http://arxiv.org/pdf/1506.06153.pdf. The
number I reported for LazEv was using 5th order dissipation applied
to all 21 evolved variables (alp, beta^i, At_{ij}, gt_{ij},
Gammat^{i}, trK, and xi).
LazEv is calculating the Hamiltonian and Momentum constraints as
well as the constraints on Gammat^i using 4th order stencils.
I also forgot to mention that I did these tests on 16 nodes, and the
memory usage was relatively low ( ~2 GB/MPI process ).
I will give the internal dissipation for McLachlan a try.
Thanks,
Jim
On 07/27/2015 05:56 PM, Erik Schnetter wrote:
Jim
Thanks for posting the details.
Can you give us more details about the LazEv scheme? In particular
there may be differences in the gauge. What (gauge) variables do
you evolve? What gauge conditions do you use? And what kind of
dissipation do you apply? Can you point us to the source code?
For the new McLachlan, you would probably use the built-in
dissipation instead of thorn Dissipation, which should lead to a
small speed-up.
-erik
On Mon, Jul 27, 2015 at 4:43 PM, James Healy <jchsma@rit.edu2) Linux 4) intel/13.0.2.146 <http://13.0.2.146> 6)<mailto:jchsma@rit.edu>> wrote:
Hello,
I have been running some tests on Stampede comparing the run
speed of McLachlan to RIT's evolution thorn LazEv. I started
with the qc0-mclachlan.par parameter file included with the
Einstein Toolkit, added a few refinement levels, increased the
resolution and changed McLachlan to be 8th order (and
increased the number of ghost zones to 5). I also increased
the initial separation so the finest grids aren't already
overlapping. To compare with LazEv, I removed the McLachlan
and Dissipation thorns and replaced them with LazEv.
Everything else in the parameter file is exactly the same. I
tried using both the McLachlan master and rewrite branches.
The grid setup is 10 levels of refinement, dx=4M on the
coarsest with outer boundary at 400M, M/128 on the finest with
r=0.6M, CFL is 0.25. Both use 8th order spatial differencing
with ghost_size=5 and 5th order dissipation.
Below is a summary of the results as reported at iteration 256
from Carpet::physical_time_per_hour:
McLachlan - rewrite branch: 3.0596110 M/hr
McLachlan - master branch: 3.8033607 M/hr
LazEv - 4.1941544 M/hr
I am using the stampede-impi.cfg configuration file in
simfactory. "module list" returns:
1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7) TACC
cluster 8) impi/4.1.0.030 <http://4.1.0.030>Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org>
Attached is my parameter file. I pasted the McLachlan
parameters below. Are there any optimizations that I can use
for McLachlan? Are the parameters I am using for it what would
be used for production runs?
ML_BSSN::harmonicN = 1 # 1+log
ML_BSSN::harmonicF = 2.0 # 1+log
ML_BSSN::ShiftGammaCoeff = 0.75
ML_BSSN::BetaDriver = 1.0
ML_BSSN::LapseAdvectionCoeff = 1.0
ML_BSSN::ShiftAdvectionCoeff = 1.0
ML_BSSN::MinimumLapse = 1.0e-8
ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::my_rhs_boundary_condition = "NewRad"
Boundary::radpower = 2
ML_BSSN::ML_log_confac_bound = "none"
ML_BSSN::ML_metric_bound = "none"
ML_BSSN::ML_Gamma_bound = "none"
ML_BSSN::ML_trace_curv_bound = "none"
ML_BSSN::ML_curv_bound = "none"
ML_BSSN::ML_lapse_bound = "none"
ML_BSSN::ML_dtlapse_bound = "none"
ML_BSSN::ML_shift_bound = "none"
ML_BSSN::ML_dtshift_bound = "none"
ML_BSSN::fdOrder = 8
ActiveThorns = "Dissipation"
Dissipation::order = 5
Dissipation::vars = "
ML_BSSN::ML_metric
ML_BSSN::ML_trace_curv
ML_BSSN::ML_curv
ML_BSSN::ML_Gamma
ML_BSSN::ML_lapse
ML_BSSN::ML_shift
ML_BSSN::ML_dtlapse
ML_BSSN::ML_dtshift
"
ActiveThorns = "ML_ADMConstraints"
Thanks,
Jim Healy
_______________________________________________
Users mailing list
http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu>>
http://www.perimeterinstitute.ca/personal/eschnetter/
--
Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu>>
http://www.perimeterinstitute.ca/personal/eschnetter/
_______________________________________________
Users mailing list
Users@einsteintoolkit.org
http://lists.einsteintoolkit.org/mailman/listinfo/users
--
Dr. Yosef Zlochower
Center for Computational Relativity and Gravitation
Associate Professor
School of Mathematical Sciences
Rochester Institute of Technology
85 Lomb Memorial Drive
Rochester, NY 14623
Office:74-2067
Phone: +1 585-475-6103
yosef@astro.rit.edu
CONFIDENTIALITY NOTE: The information transmitted, including
attachments, is intended only for the person(s) or entity to which it
is addressed and may contain confidential and/or privileged material.
Any review, retransmission, dissemination or other use of, or taking
of any action in reliance upon this information by persons or entities
other than the intended recipient is prohibited. If you received this
in error, please contact the sender and destroy any copies of this
information.