[Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

List overview All Threads
Download

newer

older

[Einstein Toolkit] #694: Behavior...

[Einstein Toolkit] #120: Improve...

Einstein Toolkit

15 Sep 2011 15 Sep '11

1:08 p.m.

#558: Carpet with Periodic Boundary Conditions -----------------------------+---------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: new Priority: major | Milestone: Component: Carpet | Version: Keywords: carpet periodic | -----------------------------+---------------------------------------------- The current Carpet driver is not compatible with the periodic boundary conditions (as provided by the Periodic thorn or otherwise).

Erik's comment on the User's list was: "It is straightforward to implement periodicity in the grid structure -- one needs to take the current grid structure, shift it in the 26 directions, and take the logical union of all 27 grid structures. This will then be automatically clipped. The most complex part is calculating by how much to shift."

Please implement support for periodic BCs in Carpet. This will directly enable production science runs.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

15 Sep 15 Sep

2:41 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: review Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+--------------------------------------------- Changes (by eschnett):

* status: new => review

Comment:

Please try the attached patch.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:03 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by hfinkel@…):

Unfortunately, that does not seem to work. I tried setting: CarpetRegrid2::symmetry_periodic_x = yes CarpetRegrid2::symmetry_periodic_y = yes CarpetRegrid2::symmetry_periodic_z = yes and with and without Periodic::periodic = yes. When the regridding meets the boundary, odd values appear.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

16 Sep 16 Sep

9:41 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by hfinkel@…):

It looks like something is wrong with the testing condition, because even when the regridding meets the boundary, I never see the "making regions periodic in the..." message with veryverbose turned on.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

12:49 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by eschnett):

I believe the offset (integer domain size) is calculated wrong. I attach a new patch.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4:11 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by hfinkel@…):

Unfortunately, the new patch also does not seem to work. Also with this version, I never see the "making regions periodic in the..." message with veryverbose turned on.

I'll post my stdout.log file.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:26 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by eschnett):

Next attempt; the offset was calculated wrong.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

18 Sep 18 Sep

2:15 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by hfinkel@…):

We seem to be getting closer. Do we need to insert a boundary_clip() operation after the fixup in each direction? I ask because... Refinement level 1: making regions periodic in the x direction... New regions are bboxset<CCTK_INT4,3>:{size=164454,setsize=63,set=[([... Refinement level 1: making regions periodic in the y direction... New regions are bboxset<CCTK_INT4,3>:{size=493362,setsize=189,set=[([

and it is still running after many hours, seemingly stuck while executing: CarpetRegrid2::periodic<1>::test_impl(...), specifically executing the line: ibset const difference = symmetrised - regions.AT(rl); (so this is in the post-enforcement check for the y direction)

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:44 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by hfinkel@…):

It did eventually move on to doing CarpetRegrid2::periodic<2>::test_impl, so it was not in an endless loop, but it then started processing a box set with some 1.8k members.

If I understand the problem correctly, then inserting extra calls to: properties.push_back (new boundary_clip()); after the queueing of periodic<0>() and periodic<1>(), should help, but that does not seem like a general solution (because it implies extra overhead even when the periodic b.c. handing is turned off).

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

19 Sep 19 Sep

8:54 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by eschnett):

Inserting an extra call to clip should not be a performance issue, because these routines are only called during regridding, not during evolution.

One alternative may be to modify the shifting to

// Shift boxes upwards and downwards by one period symmetrised |= bb.shift(+ioffset / bb.stride()) & baseextent; symmetrised |= bb.shift(-ioffset / bb.stride()) & baseextent;

which does part of what the clipping does.

The underlying problem, of course, is Carpet's inefficient implementation of bboxsets. This really needs to improve.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:56 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

Comment (by hfinkel@…):

Inserting the extra calls to: properties.push_back (new boundary_clip()); does allow the regridding to proceed in a reasonable amount of time, but, unfortunately, something is still wrong. I've attached new image from right after it regrids on the boundary. As you can see, the grid now does seem periodic, but bogus data seems nevertheless to be copied into the level 1 cells.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:10 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 Nov 8 Nov

8:42 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+--------------------------------------------- Changes (by knarf):

* status: review => reopened

Comment:

As long as something is wrong with the patch, it should probably not be up for review.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:11 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

25 Nov 25 Nov

10:59 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by bentivegna):

I would also be interested in getting this to work. I've tried to reproduce the problem using a slight variant of ks-mclachlan.par (attached). It seems that the problem is not connected to the new patch, which as far as I understand seems correct, but to some other problem in Carpet which emerges when the refinements are touching the outer boundaries. In brief, some boundary points are poisoned starting from the first iteration.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:12 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

26 Nov 26 Nov

7:14 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by eschnett):

I tried this parameter file, and I did not see any nans. How many processes do you use? Which points of which variables contain poison at what iteration?

I do see a problem with the reduction weight that seems to be set up wrong.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:13 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

27 Nov 27 Nov

12:42 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by eschnett):

I've attached a parameter file that uses cell centering instead of vertex centering. This is more natural for periodic boundaries, because upper and lower boundary can be treated in the same way. Can you try this parameter file?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:14 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

28 Nov 28 Nov

6:10 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by bentivegna):

I see the same behavior with the CC parfile. I'm running with one process, one thread.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:15 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:25 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+--------------------------------------------- Changes (by bentivegna):

* cc: bentivegna (added)

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:16 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:01 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by eschnett):

Eloisa, you mentioned poison. Poison is much easier to verify and debug than looking at png images. Can you tell me when and where you are seeing poison, and how many processes you use?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:17 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

1 Dec 1 Dec

8:17 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by bentivegna):

Indeed, it is poison. I've now realized that the issue with periodic boundaries is not the same as the one that arises on actual physical boundaries, reported above. I'm now concentrating on the issue with periodic boundaries: the poison is on all the boundary points of refinement levels with rl>0. I'm looking into thorn Periodic.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:18 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

9:01 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by hinder):

Specifically, it seems to be a problem with the Gammas, A, B, etc, (which are initialised using finite differencing) on the UPPER intersection of the symmetry and refinement boundary. This seems very similar to what happens with multipatch, where the intersection of the outer and interpatch (symmetry) boundary is not initialised. We said this was due to the order in which things happen.

So here we are seeing problems with the intersection of refinement boundaries and either symmetry or outer boundaries (probably two separate issues), and with multipatch we see problems with the intersection of symmetry and outer boundaries, to complete the set!

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:19 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

2:11 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by eschnett):

I don't see poison nor nans in my runs. Can you attach an output file with that problem?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:20 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

5 Dec 5 Dec

4:26 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by bentivegna):

Please find the 1D output of ml_gamma below. Please notice that the poison can occur both on the upper and on the lower faces; I've slightly modified the parameter file to show that.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:21 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4:29 a.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by bentivegna):

Erik, could you give me a location where I can find the Formaline tars for your setup? Maybe this is the fastest way to figure out what's different.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:22 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 Dec 8 Dec

3:08 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: reopened Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by bentivegna):

I've looked into this in more depth and I've found at least two major issues that prevent the original setup from working correctly:

1) Slab (used by Periodic for filling the symmetry boundaries) doesn't seem to work when moving data between two components of the grid that are owned by the same process (this is also stated in the documentation); as a result, when a refinement level is split into two components (as would have to be the case for one that overflows on one side of the domain and has a chuck appearing on the opposite face) and they are both handled by the same process, the boundaries are simply not filled. If these points existed when the initial data was set, they keep those values forever. If the refinement appears later, it will contain poison. This problem can be reproduced with ks-mclachlan.par, and I'll create a new ticket for Slab; 2) The original example also tries using true AMR. I'm quite unfamiliar on how this should be done, and would like to understand exactly at which point the level_mask needs to be set. With the current location (preregrid), the grid structure will be different depending on whether poisoning is used, which seems to point out that something is used before the user thorn has the chance to set it. Notice that the question is complicated by the fact that the level_mask is set from a derivative of a grid function, so that it will take a few operations (interior loop, syncs, boundary conditions, …) to fill the whole domain with the correct values. The ET list is probably a good place for this discussion.

Regarding the original request of making Carpet aware of periodic boundaries, as far as I can test Erik's third patch works and closes this ticket.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:23 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: closed Priority: major | Milestone: Component: Carpet | Version: Resolution: fixed | Keywords: carpet periodic ------------------------------+--------------------------------------------- Changes (by eschnett):

* status: reopened => closed * resolution: => fixed

Comment:

ad 1: Yes, this is correct. This used to not be a problem (in 2003), and comes back to bite now. Apologies for not implementing a test for this -- all other symmetry thorns do have such a test. There is now a ticket for this, #694.

ad 2: PreRegrid is the place. Let's discuss on the list.

Closing the ticket since the original issue seems to be solved.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:24 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 Feb 8 Feb

1:10 p.m.

New subject: [Einstein Toolkit] #558: Carpet with Periodic Boundary Conditions

#558: Carpet with Periodic Boundary Conditions ------------------------------+--------------------------------------------- Reporter: hfinkel@… | Owner: eschnett Type: enhancement | Status: closed Priority: major | Milestone: Component: Carpet | Version: Resolution: fixed | Keywords: carpet periodic ------------------------------+---------------------------------------------

Comment (by eschnett):

Applied.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/558#comment:25 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

5356

Age (days ago)

5502

Last active (days ago)

trac@lists.einsteintoolkit.org

25 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit