Erik,
The problem occurs with just one process.
Thanks again, Hal
On Tue, 2011-09-13 at 18:45 -0400, Erik Schnetter wrote:
Hal
Were you running with multiple processes?
-erik
On Tue, Sep 13, 2011 at 12:00 PM, Hal Finkel hfinkel@anl.gov wrote:
Ian,
I'd not found the problem yet. Hopefully this will be the hint we need to get this fixed.
Thanks again, Hal
On Tue, 2011-09-13 at 15:31 +0100, Ian Hawke wrote:
Erik, Hal,
Did you have any luck tracking down this error? I've just come back to this and am seeing the same error message; it appears to arise when two grids on a refined level merge, as in:
[3][0][0] exterior: [-0.010000,-0.020000,-0.020000] :[0.000000,0.020000,0.020000] : [0.001250,0.001250,0.001250] [3][0][1] exterior: [0.010000,-0.020000,-0.020000] : [0.040000,0.020000,0.020000] : [0.001250,0.001250,0.001250]
becomes
[3][0][0] exterior: [-0.010000,-0.020000,-0.020000] : [0.040000,0.020000,0.020000] : [0.001250,0.001250,0.001250]
It seems that the old grids are destroyed before the data is copied/populated in Recompose (either that or the old grid structure is not referred to in the data transfer).
Ian
On 07/09/11 01:03, Erik Schnetter wrote:
Hal
This is where numbers are assigned to components. The communication schedule decides which component needs to send data to which other component (which may be located on another process or not); this schedule is created for each refinement level independently, and may (if there is an error) refer to component numbers that don't exist. This schedule is set up in dh.cc.
Can you send me the example you are currently running (your source code and parameter file)? I will try to give it a try.
-erik
On Tue, Sep 6, 2011 at 7:44 PM, Hal Finkelhfinkel@anl.gov wrote:
On Thu, 2011-09-01 at 15:16 -0400, Erik Schnetter wrote:
On Thu, Sep 1, 2011 at 3:05 PM, Hal Finkelhfinkel@anl.gov wrote: > On Thu, 2011-09-01 at 14:25 -0400, Erik Schnetter wrote: >> On Thu, Sep 1, 2011 at 11:51 AM, Hal Finkelhfinkel@anl.gov wrote: >>> On Thu, 2011-09-01 at 11:37 -0400, Erik Schnetter wrote: >>>> On Thu, Sep 1, 2011 at 10:53 AM, Hal Finkelhfinkel@anl.gov wrote: >>>>> On Tue, 2011-08-30 at 21:06 -0400, Erik Schnetter wrote: >>>>>> On Tue, Aug 30, 2011 at 5:28 PM, Hal Finkelhfinkel@anl.gov wrote: >>>>>>> Could I also decrease the block size? I currently have >>>>>>> CarpetRegrid2::adaptive_block_size = 4, could it be smaller than that? >>>>>>> Is there a restriction based on the number of ghost points? >>>>>> Yes, you can reduce the block size. I assume that both the regridding >>>>>> operation and the time evolution will become slower if you do that, >>>>>> because more blocks will have to be handled. >>>>> Regardless of what I do, once we get past the first coarse time step, >>>>> the program seems to "hang" at "INFO (Carpet): [ml=0][rl=0][m=0][tl=0] >>>>> Regridding map 0...". >>>>> >>>>> Overall, it is in dh::regrid(do_init=true). It spends most of its time >>>>> in bboxset<int, 3>::normalize() and, specifically, mostly in the loop: >>>>> for (typename bset::iterator nsi = nbs.begin(); nsi != nbs.end(); ++ >>>>> nsi). The normalize() function does exit, however, so it is not hanging >>>>> in that function. >>>>> >>>>> The core problem seems to be that it takes a long time to execute: >>>>> boxes = boxes .shift(-dir) - boxes; >>>>> in dh::regrid(do_init=true). Probably because boxes has 129064 elements. >>>>> The coarse grid is now only 30^3 and I've left the regrid box size at 4. >>>>> I'd think, then, that the coarse grid should have a maximum of 30^3/4^3 >>>>> ~ 420 refinement regions. >>>>> >>>>> What is the best way to figure out what is going on? >>>> Hal >>>> >>>> Yes, this function is very slow. I did not expect it to be >>>> prohibitively slow. Are you compiling with optimisation enabled? >>> I've tried with optimizations enabled (and without for debugging). >>> >>>> The bboxset represents the set of refined regions, and it is >>>> internally represented as a list of bboxes (regions). Carpet performs >>>> set operations on these (intersection, union, complement, etc.) to >>>> determine the communication schedule, i.e. which ghost zones of which >>>> bbox need to be filled from which other bbox. Unfortunately, the >>>> algorithm used for this is O(n^2) in the number of refined regions, >>>> and set operations when implemented via lists themselves are O(n^2) in >>>> the set size, leading to a rather unfortunate overall complexity. The >>>> only cure is to reduce the number of bboxes (make them larger) and to >>>> regrid fewer times. >>> This is what I suspected, but nevertheless, is there something wrong? >>> How many boxes do you expect that I should have? The reason that it does >>> not finish, even with optimizations, is that there are 129K boxes in the >>> loop (that's at least 16 billion box normalizations?). >>> >>> The coarse grid is only 30^3, and the regrid box size is 4, so at >>> maximum, there should be ~400 level one boxes. Even if some of those >>> have level 2 boxes, I don't understand how there could be 129K boxes. >> The refinement structure itself should have one bbox per refined 4^3 >> box, and both CarpetRegrid2 and CarpetLib would try to combine these >> into fewer boxes where possible, i.e. where one can form rectangles or >> larger cubes. I would thus expect no more than (30/4)^2 = 64 bboxes on >> level one. > That makes sense. I think that there is a bug somewhere which is causing > the box set to be much too big. Furthermore, it does not happen on every > run, only sometimes. When it does not happen, I hit another bug after a > few coarse timesteps: > > I get a range-check exception from std::vector in a call to: > gh::get_local_component (rl=1, c=8) > the problem is that this returns: > local_components_.AT(rl).AT(c); > and local_components_[1].size() is 8 > The call to get_local_component is coming from ggf::transfer_from_all > at: > int const lc2 = h.get_local_component(rl2,c2); > where c2 is from psend.component. > So it looks like there is an off-by-one error somewhere. Very strange. This code should be quite solid by now. psend is set in the file dh.cc in thorn Carpet/CarpetLib; there is one (large) routine that calculates the communication schedule. Some of the indexing errors there in the past included confusing the number of components on different refinement levels, which led to indexing errors such as the one you describe.
The bad component numbers are not coming from: preg.component = tmpncomps.AT(m)++; in Carpet/src/Recompose.cc
Where else are the component numbers assigned?
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Hal Finkel Postdoctoral Appointee Leadership Computing Facility Argonne National Laboratory 1-630-252-0023 hfinkel@anl.gov
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users