[Users] boundary conditions with Llama
Miguel Zilhão
miguel.zilhao.nogueira at tecnico.ulisboa.pt
Wed Jun 26 04:19:52 CDT 2019
hi Erik,
i've made a gist here: https://gist.github.com/mzilhao/6eadbee1abd6ffb04faa1857e84d72a8
i've added the code snippet and (some of) the corresponding output from processor 2, as well as the
parfile used. this parfile uses some private thorns, but i think the only relevant thing is the grid
structure and coordinates. also, this used 48 processors.
is there anything else i should provide?
thanks,
Miguel
On 25/06/19 19:59, Erik Schnetter wrote:
> Miguel
>
> I do not know what you see, thus I cannot comment. Can you post your
> output/results somewhere, e.g. on "gist.github.com"?
>
> -erik
>
> On Tue, Jun 25, 2019 at 2:53 PM Miguel Zilhão
> <miguel.zilhao.nogueira at tecnico.ulisboa.pt> wrote:
>>
>> hi all,
>>
>> thanks for your replies. i will try to implement something based on these suggestions. in any case
>> though, it still seems to me that the behaviour of the Sn function from the Interpolate2 thorn is
>> not the expected one...
>>
>> i understand that some points can be both "outer boundary" and "inter-process boundary", but when
>> using an outer radius of 128, i wouldn't expect a point at radius~60 to ever be flagged as "outer
>> boundary"... yet, this does happen for some of the tests that i've performed. if this is indeed the
>> expected behaviour, maybe some clarification could be added to its interface.ccl, or documentation?
>>
>> thanks,
>> Miguel
>>
>>
>> On 25/06/19 15:24, Erik Schnetter wrote:
>>> Miguel
>>>
>>> Grid functions have edges and corners, and thus there are points that
>>> are both "outer boundary" and "inter-process boundary".
>>>
>>> The canonical way to find out where the outer boundaries are is to call
>>>
>>> # The location of the boundary points
>>> CCTK_INT FUNCTION MultiPatch_GetBoundarySpecification \
>>> (CCTK_INT IN map, \
>>> CCTK_INT IN size, \
>>> CCTK_INT OUT ARRAY nboundaryzones, \
>>> CCTK_INT OUT ARRAY is_internal, \
>>> CCTK_INT OUT ARRAY is_staggered, \
>>> CCTK_INT OUT ARRAY shiftout)
>>>
>>> This specifies, for a given map, how many boundary points there are
>>> (nboundaryzones).
>>>
>>> You also call
>>>
>>> CCTK_INT FUNCTION \
>>> MultiPatch_GetBbox \
>>> (CCTK_POINTER_TO_CONST IN cctkGH, \
>>> CCTK_INT IN size, \
>>> CCTK_INT OUT ARRAY bbox)
>>>
>>> to find out whether each of the 6 boundaries are outer boundaries.
>>> "Outer boundary" here means that you need to apply an outer boundary
>>> condition. Note that this might include some ghost zones. Depending on
>>> your setup, you can either synchronize before applying outer boundary
>>> conditions, and then apply them on ghost zones as well, or you can
>>> skip applying outer boundary conditions on outer boundary points that
>>> are ghost points, and then synchronize later.
>>>
>>> To find out which points are ghost points, look at cctk_bbox in cctkGH.
>>>
>>> -erik
>>>
>>>
>>> On Tue, Jun 25, 2019 at 9:50 AM Haas, Roland <rhaas at illinois.edu> wrote:
>>>>
>>>> Hello Miguel,
>>>>
>>>> the official way would be to use CoordBase's GetBoundarySizesAndTypes
>>>> function or replicate the code in repos/mclachlan/ML_BSSN/src/Kranc.cc,
>>>> specifically GetBoundaryInfo.
>>>>
>>>> The repsonses in the is_symbnd from GetBoundarySizesAndTypes and
>>>> GetBoundaryInfo likely differ since the former does not know about Llama so will classify Llama's inter-patch boundaries as "symmetry" boundaries while the latter does not do so.
>>>>
>>>> Their answer's for physical boundaries (which may not quite be the same
>>>> as "outer" boundaries if symmetry boundaries count as "outer") should
>>>> be identical though. And you are looking for the physical boundaries
>>>> after all.
>>>>
>>>> Yours,
>>>> Roland
>>>>
>>>>> hi Roland, all,
>>>>>
>>>>> i'm coming back to this point since i'm now realizing that my interpretation was not correct... just to recall, i'm imposing boundary conditions in the following fashion:
>>>>>
>>>>> reflevel = GetRefinementLevel(cctkGH)
>>>>> map = MultiPatch_GetMap(cctkGH)
>>>>> if (reflevel /= 0 .or. map == 0) return
>>>>>
>>>>> do j = 1, cctk_lsh(2)
>>>>> do i = 1, cctk_lsh(1)
>>>>> do k = cctk_lsh(3)-cctk_nghostzones(3)+1, cctk_lsh(3)
>>>>> if (Sn(i,j,k) == -2) then
>>>>>
>>>>> [my BCs go here]
>>>>>
>>>>> end if
>>>>> end do
>>>>> end do
>>>>> end do
>>>>>
>>>>> and my assumption was that the line "if (Sn(i,j,k) == -2)" would guarantee that this would only run in the physical boundary region, as per the comment in the file interface.ccl from llama/Interpolate2:
>>>>>
>>>>> CCTK_INT source_patch TYPE=gf TAGS='Checkpoint="no" Prolongation="none"'
>>>>> {
>>>>> Sn
>>>>> } "source patch number, -1 for interior points, -2 for outer boundary points, -3 for inter-
>>>>> processor ghost points"
>>>>>
>>>>> this interpretation was consistent with what it's done in the thorn LlamaWaveToy (where the BC are imposed in a similar way).
>>>>>
>>>>> however, when running with several processors, it seems that there are some inter-processor points marked with Sn = -2, and therefore the loop above is called outside my physical boundary region (i noticed this by adding write statements inside the loop above).
>>>>>
>>>>> so i have the following questions:
>>>>>
>>>>> - what exactly does "outer boundary points" mean for the Sn function? is there a bug, or did i just misinterpreted that "outer boundary" means the "physical" boundary?
>>>>>
>>>>> - what would be the canonical way of checking whether a point belongs to the physical outer boundary (ie, where BCs should be specified) when using Llama?
>>>>>
>>>>> thanks,
>>>>> Miguel
>>>>>
>>>>>
>>>>> On 18/06/19 16:00, Haas, Roland wrote:
>>>>>> Hello Miguel,
>>>>>>
>>>>>>> oh, but won't this be handled by the
>>>>>>>
>>>>>>> if (Sn(i,j,k) == -2)
>>>>>>>
>>>>>>> line above?
>>>>>> I see. I had no idea that this is what the grid function does.
>>>>>>
>>>>>>> from the description of the Sn function in the Interpolate2 thorn:
>>>>>>>
>>>>>>> CCTK_INT source_patch TYPE=gf TAGS='Checkpoint="no"
>>>>>>> Prolongation="none"' {
>>>>>>> Sn
>>>>>>> } "source patch number, -1 for interior points, -2 for outer
>>>>>>> boundary points, -3 for inter- processor ghost points"
>>>>>>>
>>>>>>> and also from the way this is done in LlamaWaveToy, i was assuming
>>>>>>> that Sn(i,j,k) == -2 would guarantee that the points belong to the
>>>>>>> outer boundary (ie, points where i want to specify a physical
>>>>>>> boundary condition). is this assumption not correct?
>>>>>> Yes, in that case this should work fine. In fact it is probably more
>>>>>> correct then what I did b/c due to the way Llama works may suggestion
>>>>>> would likely have treated the inter-patch bondaries in Llama as outer
>>>>>> boundaries (since they are "symmetry" boundaries to Cactus).
>>>>>>
>>>>>> So you should be fine actually.
>>>>>>
>>>>>> Yours,
>>>>>> Roland
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> My email is as private as my paper mail. I therefore support encrypting
>>>> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at einsteintoolkit.org
>>>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>>
>>>
>>>
>
>
>
More information about the Users
mailing list