[Users] Carpet::processor_topology="recursive"...

[Erik Schnetter]

Yosef

A heads up: Peter Diener and I are just debugging this. While the new
load balancing mechanism seems to work for simple domains, there are
apparently problems with more complex topologies.

-erik

On Thu, Nov 17, 2011 at 9:06 AM, Yosef Zlochower <yosef at astro.rit.edu> wrote:
> Removing rotating symmetry fixes this error, but then the code
> segfaults. If I also remove the new topology option, the code
> seems to run.
>
> On 11/17/2011 06:10 AM, Yosef Zlochower wrote:
>>
>> Hi,
>>
>>   I am trying to test the new balancing algorithm with LazEv (I am
>> using the development version of ET).
>> I used an identical test parfile to the one I used for Maxwell and
>> previous ET versions (except for the
>> Carpet::processor_topology="recursive" parameter). On running, I
>> get a grid setup error
>>
>> The following components are not properly nested, i.e.,
>> they are not contained within the next coarser level's components:
>>     ml 0 rl 7 c 7:
>>
>> ([88,25480,1016]:[1416,25720,1144]:[4,4,4]/[22,6370,254]:[354,6430,286]/[333,61,33]/670329)
>> WARNING level 0 in thorn CarpetLib processor 2 host
>> i148-303.ranger.tacc.utexas.edu
>>    (line 191 of
>>
>> /work/00951/tg801986/ET/Latest/Cactus/arrangements/Carpet/CarpetLib/src/gh.cc):
>>    ->  The refinement hierarchy is not properly nested.
>> [2] [MPI Abort by user] Aborting Program!
>> WARNING level 0 in thorn CarpetLib processor 6 host
>> i176-108.ranger.tacc.utexas.edu
>>    (line 191 of
>>
>> /work/00951/tg801986/ET/Latest/Cactus/arrangements/Carpet/CarpetLib/src/gh.cc):
>>    ->  The refinement hierarchy is not properly nested.
>>
>> Here is my grid setup
>> ActiveThorns = .... ReflectionSymmetry RotatingSymmetry180 ....
>>
>> CoordBase::domainsize       = "minmax"
>> CoordBase::xmin             = 0
>> CoordBase::ymin             = -400
>> CoordBase::zmin             = 0
>> CoordBase::xmax             = 400
>> CoordBase::ymax             = 400
>> CoordBase::zmax             = 400
>> CoordBase::spacing          = "numcells"
>> CoordBase::ncells_x         = 100
>> CoordBase::ncells_y         = 200
>> CoordBase::ncells_z         = 100
>> CartGrid3D::type            = "coordbase"
>> CartGrid3D::domain          = "full"
>>
>> CartGrid3D::avoid_originx   = no
>> CartGrid3D::avoid_originy   = no
>> CartGrid3D::avoid_originz   = no
>>
>> CoordBase::boundary_size_x_lower        = 4
>> CoordBase::boundary_size_y_lower        = 1
>> CoordBase::boundary_size_z_lower        = 4
>>
>> CoordBase::boundary_size_x_lower        = 4
>> CoordBase::boundary_size_y_lower        = 1
>> CoordBase::boundary_size_z_lower        = 4
>> CoordBase::boundary_shiftout_x_lower    = 1
>> CoordBase::boundary_shiftout_y_lower    = 0
>> CoordBase::boundary_shiftout_z_lower    = 1
>>
>> CoordBase::boundary_size_x_upper        = 1
>> CoordBase::boundary_size_y_upper        = 1
>> CoordBase::boundary_size_z_upper        = 1
>>
>> ReflectionSymmetry::reflection_x="no"
>> ReflectionSymmetry::reflection_y="no"
>> ReflectionSymmetry::reflection_z="yes"
>> ReflectionSymmetry::avoid_origin_x="no"
>> ReflectionSymmetry::avoid_origin_y="no"
>> ReflectionSymmetry::avoid_origin_z="no"
>> ReflectionSymmetry::avoid_origin_upper_x="no"
>> ReflectionSymmetry::avoid_origin_upper_y="no"
>> ReflectionSymmetry::avoid_origin_upper_z="no"
>>
>>
>> driver::ghost_size                        = 4
>>
>> CarpetRegrid2::regrid_every = 32
>>
>> CarpetRegrid2::num_centres = 2
>> CarpetRegrid2::symmetry_rotating180 = yes
>>
>> CarpetRegrid2::num_levels_1 = 9
>> CarpetRegrid2::position_x_1 = 4.250
>> CarpetRegrid2::position_y_1 = 0
>> CarpetRegrid2::position_z_1 = 0
>> CarpetRegrid2::radius_1[ 1] = 208
>> CarpetRegrid2::radius_1[ 2] = 115
>> CarpetRegrid2::radius_1[ 3] = 60
>> CarpetRegrid2::radius_1[ 4] = 30
>> CarpetRegrid2::radius_1[ 5] = 12
>> CarpetRegrid2::radius_1[ 6] = 5
>> CarpetRegrid2::radius_1[ 7] = 1.5
>> CarpetRegrid2::radius_1[ 8] = 0.75
>>
>> CarpetRegrid2::num_levels_2 = 9
>> CarpetRegrid2::position_x_2 = -4.250
>> CarpetRegrid2::position_y_2 = 0
>> CarpetRegrid2::position_z_2 = 0
>> CarpetRegrid2::radius_2[ 1] = 208
>> CarpetRegrid2::radius_2[ 2] = 115
>> CarpetRegrid2::radius_2[ 3] = 60
>> CarpetRegrid2::radius_2[ 4] = 30
>> CarpetRegrid2::radius_2[ 5] = 12
>> CarpetRegrid2::radius_2[ 6] = 5
>> CarpetRegrid2::radius_2[ 7] = 1.5
>> CarpetRegrid2::radius_2[ 8] = 0.75
>>
>>
>> On 11/14/2011 04:15 PM, Erik Schnetter wrote:
>>>
>>> ... is the way to enable Peter Dieners new load balancing algorithm
>>> that I just adapted to Carpet. It should reduce load imbalance. If you
>>> feel adventurous, please give it a try, and compare to an existing
>>> simulation.
>>>
>>> I am in particular interested in knowing
>>> - does this break stuff?
>>> - do the simulation results remain the same?
>>> - how does the speed compare to the standard load balancing algorithm?
>>>
>>> -erik
>>>
>>
>>
>
>
> --
> Dr. Yosef Zlochower
> Center for Computational Relativity and Gravitation
> Assistant Professor
> School of Mathematical Sciences
> Rochester Institute of Technology
> 85 Lomb Memorial Drive
> Rochester, NY 14623
>
> Office:74-2067
> Phone: +1 585-475-6103
>
> yosef at astro.rit.edu
>
> CONFIDENTIALITY NOTE: The information transmitted, including
> attachments, is intended only for the person(s) or entity to which it
> is addressed and may contain confidential and/or privileged material.
> Any review, retransmission, dissemination or other use of, or taking
> of any action in reliance upon this information by persons or entities
> other than the intended recipient is prohibited. If you received this
> in error, please contact the sender and destroy any copies of this
> information.
>



-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/