Yosef

A heads up: Peter Diener and I are just debugging this. While the new load balancing mechanism seems to work for simple domains, there are apparently problems with more complex topologies.

-erik

On Thu, Nov 17, 2011 at 9:06 AM, Yosef Zlochower yosef@astro.rit.edu wrote:

Removing rotating symmetry fixes this error, but then the code segfaults. If I also remove the new topology option, the code seems to run.

On 11/17/2011 06:10 AM, Yosef Zlochower wrote:

...

Hi,

I am trying to test the new balancing algorithm with LazEv (I am using the development version of ET). I used an identical test parfile to the one I used for Maxwell and previous ET versions (except for the Carpet::processor_topology="recursive" parameter). On running, I get a grid setup error

The following components are not properly nested, i.e., they are not contained within the next coarser level's components:     ml 0 rl 7 c 7:

([88,25480,1016]:[1416,25720,1144]:[4,4,4]/[22,6370,254]:[354,6430,286]/[333,61,33]/670329) WARNING level 0 in thorn CarpetLib processor 2 host i148-303.ranger.tacc.utexas.edu    (line 191 of

/work/00951/tg801986/ET/Latest/Cactus/arrangements/Carpet/CarpetLib/src/gh.cc):    ->  The refinement hierarchy is not properly nested. [2] [MPI Abort by user] Aborting Program! WARNING level 0 in thorn CarpetLib processor 6 host i176-108.ranger.tacc.utexas.edu    (line 191 of

/work/00951/tg801986/ET/Latest/Cactus/arrangements/Carpet/CarpetLib/src/gh.cc):    ->  The refinement hierarchy is not properly nested.

Here is my grid setup ActiveThorns = .... ReflectionSymmetry RotatingSymmetry180 ....

CoordBase::domainsize       = "minmax" CoordBase::xmin             = 0 CoordBase::ymin             = -400 CoordBase::zmin             = 0 CoordBase::xmax             = 400 CoordBase::ymax             = 400 CoordBase::zmax             = 400 CoordBase::spacing          = "numcells" CoordBase::ncells_x         = 100 CoordBase::ncells_y         = 200 CoordBase::ncells_z         = 100 CartGrid3D::type            = "coordbase" CartGrid3D::domain          = "full"

CartGrid3D::avoid_originx   = no CartGrid3D::avoid_originy   = no CartGrid3D::avoid_originz   = no

CoordBase::boundary_size_x_lower        = 4 CoordBase::boundary_size_y_lower        = 1 CoordBase::boundary_size_z_lower        = 4

CoordBase::boundary_size_x_lower        = 4 CoordBase::boundary_size_y_lower        = 1 CoordBase::boundary_size_z_lower        = 4 CoordBase::boundary_shiftout_x_lower    = 1 CoordBase::boundary_shiftout_y_lower    = 0 CoordBase::boundary_shiftout_z_lower    = 1

CoordBase::boundary_size_x_upper        = 1 CoordBase::boundary_size_y_upper        = 1 CoordBase::boundary_size_z_upper        = 1

ReflectionSymmetry::reflection_x="no" ReflectionSymmetry::reflection_y="no" ReflectionSymmetry::reflection_z="yes" ReflectionSymmetry::avoid_origin_x="no" ReflectionSymmetry::avoid_origin_y="no" ReflectionSymmetry::avoid_origin_z="no" ReflectionSymmetry::avoid_origin_upper_x="no" ReflectionSymmetry::avoid_origin_upper_y="no" ReflectionSymmetry::avoid_origin_upper_z="no"

driver::ghost_size                        = 4

CarpetRegrid2::regrid_every = 32

CarpetRegrid2::num_centres = 2 CarpetRegrid2::symmetry_rotating180 = yes

CarpetRegrid2::num_levels_1 = 9 CarpetRegrid2::position_x_1 = 4.250 CarpetRegrid2::position_y_1 = 0 CarpetRegrid2::position_z_1 = 0 CarpetRegrid2::radius_1[ 1] = 208 CarpetRegrid2::radius_1[ 2] = 115 CarpetRegrid2::radius_1[ 3] = 60 CarpetRegrid2::radius_1[ 4] = 30 CarpetRegrid2::radius_1[ 5] = 12 CarpetRegrid2::radius_1[ 6] = 5 CarpetRegrid2::radius_1[ 7] = 1.5 CarpetRegrid2::radius_1[ 8] = 0.75

CarpetRegrid2::num_levels_2 = 9 CarpetRegrid2::position_x_2 = -4.250 CarpetRegrid2::position_y_2 = 0 CarpetRegrid2::position_z_2 = 0 CarpetRegrid2::radius_2[ 1] = 208 CarpetRegrid2::radius_2[ 2] = 115 CarpetRegrid2::radius_2[ 3] = 60 CarpetRegrid2::radius_2[ 4] = 30 CarpetRegrid2::radius_2[ 5] = 12 CarpetRegrid2::radius_2[ 6] = 5 CarpetRegrid2::radius_2[ 7] = 1.5 CarpetRegrid2::radius_2[ 8] = 0.75

On 11/14/2011 04:15 PM, Erik Schnetter wrote:

...

... is the way to enable Peter Dieners new load balancing algorithm that I just adapted to Carpet. It should reduce load imbalance. If you feel adventurous, please give it a try, and compare to an existing simulation.

I am in particular interested in knowing

• does this break stuff?
• do the simulation results remain the same?
• how does the speed compare to the standard load balancing algorithm?

-erik

-- Dr. Yosef Zlochower Center for Computational Relativity and Gravitation Assistant Professor School of Mathematical Sciences Rochester Institute of Technology 85 Lomb Memorial Drive Rochester, NY 14623

Office:74-2067 Phone: +1 585-475-6103

yosef@astro.rit.edu

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.