Hi Erik,

We evolve alpha and beta for the gauge variables as given in equations 7a and 7b of http://arxiv.org/pdf/1506.06153.pdf. The number I reported for LazEv was using 5th order dissipation applied to all 21 evolved variables (alp, beta^i, At_{ij}, gt_{ij}, Gammat^{i}, trK, and xi).

LazEv is calculating the Hamiltonian and Momentum constraints as well as the constraints on Gammat^i using 4th order stencils.

I also forgot to mention that I did these tests on 16 nodes, and the memory usage was relatively low ( ~2 GB/MPI process ).

I will give the internal dissipation for McLachlan a try.

Thanks, Jim

On 07/27/2015 05:56 PM, Erik Schnetter wrote:

Jim

Thanks for posting the details.

Can you give us more details about the LazEv scheme? In particular there may be differences in the gauge. What (gauge) variables do you evolve? What gauge conditions do you use? And what kind of dissipation do you apply? Can you point us to the source code?

For the new McLachlan, you would probably use the built-in dissipation instead of thorn Dissipation, which should lead to a small speed-up.

-erik

On Mon, Jul 27, 2015 at 4:43 PM, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:

Hello,

I have been running some tests on Stampede comparing the run speed
of McLachlan to RIT's evolution thorn LazEv.  I started with the
qc0-mclachlan.par parameter file included with the Einstein
Toolkit, added a few refinement levels, increased the resolution
and changed McLachlan to be 8th order (and increased the number of
ghost zones to 5).  I also increased the initial separation so the
finest grids aren't already overlapping.  To compare with LazEv, I
removed the McLachlan and Dissipation thorns and replaced them
with LazEv. Everything else in the parameter file is exactly the
same.  I tried using both the McLachlan master and rewrite branches.

The grid setup is 10 levels of refinement, dx=4M on the coarsest
with outer boundary at 400M, M/128 on the finest with r=0.6M, CFL
is 0.25.  Both use 8th order spatial differencing with
ghost_size=5 and 5th order dissipation.

Below is a summary of the results as reported at iteration 256
from Carpet::physical_time_per_hour:

McLachlan - rewrite branch: 3.0596110 M/hr
McLachlan - master branch: 3.8033607 M/hr
LazEv - 4.1941544 M/hr

I am using the stampede-impi.cfg configuration file in simfactory.
"module list" returns:

  1) TACC-paths   3) cluster-paths      5) xalt/0.4.6   7) TACC
  2) Linux        4) intel/13.0.2.146 <http://13.0.2.146>  6)
cluster      8) impi/4.1.0.030 <http://4.1.0.030>

Attached is my parameter file.  I pasted the McLachlan parameters
below.  Are there any optimizations that I can use for McLachlan?
Are the parameters I am using for it what would be used for
production runs?

ML_BSSN::harmonicN           = 1      # 1+log
ML_BSSN::harmonicF           = 2.0    # 1+log
ML_BSSN::ShiftGammaCoeff     = 0.75
ML_BSSN::BetaDriver          = 1.0
ML_BSSN::LapseAdvectionCoeff = 1.0
ML_BSSN::ShiftAdvectionCoeff = 1.0

ML_BSSN::MinimumLapse        = 1.0e-8

ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"
ML_BSSN::my_rhs_boundary_condition     = "NewRad"
Boundary::radpower                     = 2

ML_BSSN::ML_log_confac_bound = "none"
ML_BSSN::ML_metric_bound     = "none"
ML_BSSN::ML_Gamma_bound      = "none"
ML_BSSN::ML_trace_curv_bound = "none"
ML_BSSN::ML_curv_bound       = "none"
ML_BSSN::ML_lapse_bound      = "none"
ML_BSSN::ML_dtlapse_bound    = "none"
ML_BSSN::ML_shift_bound      = "none"
ML_BSSN::ML_dtshift_bound    = "none"

ML_BSSN::fdOrder = 8

ActiveThorns = "Dissipation"

Dissipation::order = 5
Dissipation::vars  = "
        ML_BSSN::ML_metric
        ML_BSSN::ML_trace_curv
        ML_BSSN::ML_curv
        ML_BSSN::ML_Gamma
        ML_BSSN::ML_lapse
        ML_BSSN::ML_shift
        ML_BSSN::ML_dtlapse
        ML_BSSN::ML_dtshift
"

ActiveThorns = "ML_ADMConstraints"


Thanks,
Jim Healy

_______________________________________________
Users mailing list
Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org>
http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

Hi Erik,

Our standard choice is to use 8th order centered differencing with 5th-order dissipation. For this test, Jim used 8th-order with upwinding and thus used 5 ghost zones (rather than 4). We use the full complement of 4*ghost buffer zones. Near the boundary we reduce the order of the stencil from 8 to 6 to 4 to 2 and on the boundary we apply Sommerfeld boundary conditions (implemented as modifications to RHS).

For the gauge, we evolve d0 alpha = -2 alpha K and dt beta = 3/4 Gamma^i (we don't introduce the auxiliary B field).

We precalculate the mixed derivatives and only calculate first derivatives and non-mixed second derivatives in the main loop. We apply dissipation to all variables, but apply it one variable at a time (i.e., we go through a full 3d loop nvars times).

The conformal factor is W = sqrt(chi).

We use the admbase variables alpha and beta and fill in kij and gij after each ministep of the RK4 integrator. We similarly enforce the algebraic constraints at each ministep. We calculate the constraints using the BSSN variables, but use a 4th order stencil and implement mixed derivatives using one dimensional derivatives along the diagonal (this is less accurate, but faster).

The FD order is a compile time option and we can test the code with and without upwinding, with FD orders of 2nd through 8, and we can use the standard or "quick" mixed derivatives for the constraint. We can similarly use 3, 5, 7, 9 order dissipation.

We use openmp parallelization along the "z" direction of our loops

On 07/27/2015 10:21 PM, Erik Schnetter wrote:

Jim

By default, McLachlan is configured to be generic, i.e. many choices can be made at run time. For example, there are 4 additional gauge variables evolved (A and B^i). If you are willing to restrict your run-time choices, for example by fixing derivative order, disabling multi-block systems, choosing a particular gauge, choice of conformal factor, etc., then this will lead to a more efficient code. You can easily do so for McLachlan by generating a new version in Kranc. If you look in the beginning of McLachlan_BSSN.m in the section "Choose code to generate", then you'll see the parameters you can choose. We have made a particular set of choices for a configuration "BSSN_bench", but these are apparently different from LazEv. For example, we use 4th order finite differencing, and evolve B^i in time as well.

Can you specify all the details of your formulation? I notice that you didn't respond to my request to point me to the source code, and there are quite a few details that you also didn't describe yet (and are also not described in your paper), such as your choice of conformal factor, whether/how your code is integrated with ADMBase, whether the constraints are calculated from the BSSN or the ADM variables, how advection derivatives are handled, etc. For a true comparison, this all needs to be known. Unless we both know exactly what both the codes do, we'll never know where the physics differences are.

-erik

On Mon, Jul 27, 2015 at 5:39 PM, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:

Hi Erik,

We evolve alpha and beta for the gauge variables as given in
equations 7a and 7b of http://arxiv.org/pdf/1506.06153.pdf.  The
number I reported for LazEv was using 5th order dissipation applied
to all 21 evolved variables (alp, beta^i, At_{ij}, gt_{ij},
Gammat^{i}, trK, and xi).

LazEv is calculating the Hamiltonian and Momentum constraints as
well as the constraints on Gammat^i using 4th order stencils.

I also forgot to mention that I did these tests on 16 nodes, and the
memory usage was relatively low ( ~2 GB/MPI process ).

I will give the internal dissipation for McLachlan a try.

Thanks,
Jim


On 07/27/2015 05:56 PM, Erik Schnetter wrote:

...

Jim

Thanks for posting the details.

Can you give us more details about the LazEv scheme? In particular
there may be differences in the gauge. What (gauge) variables do
you evolve? What gauge conditions do you use? And what kind of
dissipation do you apply? Can you point us to the source code?

For the new McLachlan, you would probably use the built-in
dissipation instead of thorn Dissipation, which should lead to a
small speed-up.

-erik

On Mon, Jul 27, 2015 at 4:43 PM, James Healy <jchsma@rit.edu
<mailto:jchsma@rit.edu>> wrote:

    Hello,

    I have been running some tests on Stampede comparing the run
    speed of McLachlan to RIT's evolution thorn LazEv.  I started
    with the qc0-mclachlan.par parameter file included with the
    Einstein Toolkit, added a few refinement levels, increased the
    resolution and changed McLachlan to be 8th order (and
    increased the number of ghost zones to 5).  I also increased
    the initial separation so the finest grids aren't already
    overlapping.  To compare with LazEv, I removed the McLachlan
    and Dissipation thorns and replaced them with LazEv.
    Everything else in the parameter file is exactly the same.  I
    tried using both the McLachlan master and rewrite branches.

    The grid setup is 10 levels of refinement, dx=4M on the
    coarsest with outer boundary at 400M, M/128 on the finest with
    r=0.6M, CFL is 0.25.  Both use 8th order spatial differencing
    with ghost_size=5 and 5th order dissipation.

    Below is a summary of the results as reported at iteration 256
    from Carpet::physical_time_per_hour:

    McLachlan - rewrite branch: 3.0596110 M/hr
    McLachlan - master branch: 3.8033607 M/hr
    LazEv - 4.1941544 M/hr

    I am using the stampede-impi.cfg configuration file in
    simfactory. "module list" returns:

      1) TACC-paths   3) cluster-paths      5) xalt/0.4.6   7) TACC
      2) Linux        4) intel/13.0.2.146 <http://13.0.2.146>  6)
    cluster      8) impi/4.1.0.030 <http://4.1.0.030>

    Attached is my parameter file.  I pasted the McLachlan
    parameters below.  Are there any optimizations that I can use
    for McLachlan? Are the parameters I am using for it what would
    be used for production runs?

    ML_BSSN::harmonicN           = 1      # 1+log
    ML_BSSN::harmonicF           = 2.0    # 1+log
    ML_BSSN::ShiftGammaCoeff     = 0.75
    ML_BSSN::BetaDriver          = 1.0
    ML_BSSN::LapseAdvectionCoeff = 1.0
    ML_BSSN::ShiftAdvectionCoeff = 1.0

    ML_BSSN::MinimumLapse        = 1.0e-8

    ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"
    ML_BSSN::my_rhs_boundary_condition     = "NewRad"
    Boundary::radpower                     = 2

    ML_BSSN::ML_log_confac_bound = "none"
    ML_BSSN::ML_metric_bound     = "none"
    ML_BSSN::ML_Gamma_bound      = "none"
    ML_BSSN::ML_trace_curv_bound = "none"
    ML_BSSN::ML_curv_bound       = "none"
    ML_BSSN::ML_lapse_bound      = "none"
    ML_BSSN::ML_dtlapse_bound    = "none"
    ML_BSSN::ML_shift_bound      = "none"
    ML_BSSN::ML_dtshift_bound    = "none"

    ML_BSSN::fdOrder = 8

    ActiveThorns = "Dissipation"

    Dissipation::order = 5
    Dissipation::vars  = "
            ML_BSSN::ML_metric
            ML_BSSN::ML_trace_curv
            ML_BSSN::ML_curv
            ML_BSSN::ML_Gamma
            ML_BSSN::ML_lapse
            ML_BSSN::ML_shift
            ML_BSSN::ML_dtlapse
            ML_BSSN::ML_dtshift
    "

    ActiveThorns = "ML_ADMConstraints"


    Thanks,
    Jim Healy

    _______________________________________________
    Users mailing list
    Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org>
    http://lists.einsteintoolkit.org/mailman/listinfo/users




--
Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu>>
http://www.perimeterinstitute.ca/personal/eschnetter/

-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

On 28 Jul 2015, at 16:56, Erik Schnetter schnetter@cct.lsu.edu wrote:

Yosef, James

I'd like to at least try to see whether the McLachlan performance you're seeing is in the right ballpark. Can you post the parameter file you're using?

You can get an idea of where time is spent by using

Carpet::output_timer_tree_every = 256

or similar. I also recommend

Carpet::sync_barriers = yes

to account for load imbalance, otherwise you might attribute time to a routine when it's really just waiting for other processors to catch up.

McLachlan is very careful to make sure that results are "correct" in a strict mesh refinement sense, and therefore does quite a lot of work in bins such as CCTK_POSTRESTRICT. In many situations, you may be able to deduce that these are unnecessary (or you may think you do not care), but they are necessary in some corner cases, and hence are enabled by default. I suggest comparing the timer trees from the two runs, and seeing which routines are different.

-erik

On Tue, Jul 28, 2015 at 9:19 AM, Yosef Zlochower yosef@astro.rit.edu wrote: Hi Erik,

Our standard choice is to use 8th order centered differencing with 5th-order dissipation. For this test, Jim used 8th-order with upwinding and thus used 5 ghost zones (rather than 4). We use the full complement of 4*ghost buffer zones. Near the boundary we reduce the order of the stencil from 8 to 6 to 4 to 2 and on the boundary we apply Sommerfeld boundary conditions (implemented as modifications to RHS).

For the gauge, we evolve d0 alpha = -2 alpha K and dt beta = 3/4 Gamma^i (we don't introduce the auxiliary B field).

We precalculate the mixed derivatives and only calculate first derivatives and non-mixed second derivatives in the main loop. We apply dissipation to all variables, but apply it one variable at a time (i.e., we go through a full 3d loop nvars times).

The conformal factor is W = sqrt(chi).

We use the admbase variables alpha and beta and fill in kij and gij after each ministep of the RK4 integrator. We similarly enforce the algebraic constraints at each ministep. We calculate the constraints using the BSSN variables, but use a 4th order stencil and implement mixed derivatives using one dimensional derivatives along the diagonal (this is less accurate, but faster).

The FD order is a compile time option and we can test the code with and without upwinding, with FD orders of 2nd through 8, and we can use the standard or "quick" mixed derivatives for the constraint. We can similarly use 3, 5, 7, 9 order dissipation.

We use openmp parallelization along the "z" direction of our loops

On 07/27/2015 10:21 PM, Erik Schnetter wrote: Jim

By default, McLachlan is configured to be generic, i.e. many choices can be made at run time. For example, there are 4 additional gauge variables evolved (A and B^i). If you are willing to restrict your run-time choices, for example by fixing derivative order, disabling multi-block systems, choosing a particular gauge, choice of conformal factor, etc., then this will lead to a more efficient code. You can easily do so for McLachlan by generating a new version in Kranc. If you look in the beginning of McLachlan_BSSN.m in the section "Choose code to generate", then you'll see the parameters you can choose. We have made a particular set of choices for a configuration "BSSN_bench", but these are apparently different from LazEv. For example, we use 4th order finite differencing, and evolve B^i in time as well.

Can you specify all the details of your formulation? I notice that you didn't respond to my request to point me to the source code, and there are quite a few details that you also didn't describe yet (and are also not described in your paper), such as your choice of conformal factor, whether/how your code is integrated with ADMBase, whether the constraints are calculated from the BSSN or the ADM variables, how advection derivatives are handled, etc. For a true comparison, this all needs to be known. Unless we both know exactly what both the codes do, we'll never know where the physics differences are.

-erik

On Mon, Jul 27, 2015 at 5:39 PM, James Healy <jchsma@rit.edu mailto:jchsma@rit.edu> wrote:

Hi Erik,

We evolve alpha and beta for the gauge variables as given in
equations 7a and 7b of http://arxiv.org/pdf/1506.06153.pdf.  The
number I reported for LazEv was using 5th order dissipation applied
to all 21 evolved variables (alp, beta^i, At_{ij}, gt_{ij},
Gammat^{i}, trK, and xi).

LazEv is calculating the Hamiltonian and Momentum constraints as
well as the constraints on Gammat^i using 4th order stencils.

I also forgot to mention that I did these tests on 16 nodes, and the
memory usage was relatively low ( ~2 GB/MPI process ).

I will give the internal dissipation for McLachlan a try.

Thanks,
Jim


On 07/27/2015 05:56 PM, Erik Schnetter wrote:
Jim

Thanks for posting the details.

Can you give us more details about the LazEv scheme? In particular
there may be differences in the gauge. What (gauge) variables do
you evolve? What gauge conditions do you use? And what kind of
dissipation do you apply? Can you point us to the source code?

For the new McLachlan, you would probably use the built-in
dissipation instead of thorn Dissipation, which should lead to a
small speed-up.

-erik

On Mon, Jul 27, 2015 at 4:43 PM, James Healy <jchsma@rit.edu
<mailto:jchsma@rit.edu>> wrote:

    Hello,

    I have been running some tests on Stampede comparing the run
    speed of McLachlan to RIT's evolution thorn LazEv.  I started
    with the qc0-mclachlan.par parameter file included with the
    Einstein Toolkit, added a few refinement levels, increased the
    resolution and changed McLachlan to be 8th order (and
    increased the number of ghost zones to 5).  I also increased
    the initial separation so the finest grids aren't already
    overlapping.  To compare with LazEv, I removed the McLachlan
    and Dissipation thorns and replaced them with LazEv.
    Everything else in the parameter file is exactly the same.  I
    tried using both the McLachlan master and rewrite branches.

    The grid setup is 10 levels of refinement, dx=4M on the
    coarsest with outer boundary at 400M, M/128 on the finest with
    r=0.6M, CFL is 0.25.  Both use 8th order spatial differencing
    with ghost_size=5 and 5th order dissipation.

    Below is a summary of the results as reported at iteration 256
    from Carpet::physical_time_per_hour:

    McLachlan - rewrite branch: 3.0596110 M/hr
    McLachlan - master branch: 3.8033607 M/hr
    LazEv - 4.1941544 M/hr

    I am using the stampede-impi.cfg configuration file in
    simfactory. "module list" returns:

      1) TACC-paths   3) cluster-paths      5) xalt/0.4.6   7) TACC
      2) Linux        4) intel/13.0.2.146 <http://13.0.2.146>  6)
    cluster      8) impi/4.1.0.030 <http://4.1.0.030>


    Attached is my parameter file.  I pasted the McLachlan
    parameters below.  Are there any optimizations that I can use
    for McLachlan? Are the parameters I am using for it what would
    be used for production runs?

    ML_BSSN::harmonicN           = 1      # 1+log
    ML_BSSN::harmonicF           = 2.0    # 1+log
    ML_BSSN::ShiftGammaCoeff     = 0.75
    ML_BSSN::BetaDriver          = 1.0
    ML_BSSN::LapseAdvectionCoeff = 1.0
    ML_BSSN::ShiftAdvectionCoeff = 1.0

    ML_BSSN::MinimumLapse        = 1.0e-8

    ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas"
    ML_BSSN::my_rhs_boundary_condition     = "NewRad"
    Boundary::radpower                     = 2

    ML_BSSN::ML_log_confac_bound = "none"
    ML_BSSN::ML_metric_bound     = "none"
    ML_BSSN::ML_Gamma_bound      = "none"
    ML_BSSN::ML_trace_curv_bound = "none"
    ML_BSSN::ML_curv_bound       = "none"
    ML_BSSN::ML_lapse_bound      = "none"
    ML_BSSN::ML_dtlapse_bound    = "none"
    ML_BSSN::ML_shift_bound      = "none"
    ML_BSSN::ML_dtshift_bound    = "none"

    ML_BSSN::fdOrder = 8

    ActiveThorns = "Dissipation"

    Dissipation::order = 5
    Dissipation::vars  = "
            ML_BSSN::ML_metric
            ML_BSSN::ML_trace_curv
            ML_BSSN::ML_curv
            ML_BSSN::ML_Gamma
            ML_BSSN::ML_lapse
            ML_BSSN::ML_shift
            ML_BSSN::ML_dtlapse
            ML_BSSN::ML_dtshift
    "

    ActiveThorns = "ML_ADMConstraints"


    Thanks,
    Jim Healy

    _______________________________________________
    Users mailing list
    Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org>
    http://lists.einsteintoolkit.org/mailman/listinfo/users




--
Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu>>
http://www.perimeterinstitute.ca/personal/eschnetter/

-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Dr. Yosef Zlochower Center for Computational Relativity and Gravitation Associate Professor School of Mathematical Sciences Rochester Institute of Technology 85 Lomb Memorial Drive Rochester, NY 14623

Office:74-2067 Phone: +1 585-475-6103

yosef@astro.rit.edu

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users