#2648: Empty ApplyBCs call in LeanBSSNMoL

Reporter: Samuel Cupp Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn

Comment (by helvi witek):

So, for full disclosure let me copy here the email conversion with Sam, Miguel and myself (apologies for email formatting):

It sounds like you would want something like LeanBSSN_Boundaries running before that. However, looking at that function I see quite a lot of grid functions in it. Do you want all of them to run? If you only want gammat you'll (unfortunately) probably need another function with just it. However, it's probably fine to apply symmetry conditions to the other grid functions (gconf_fac, hmetric, hcurv, trk, lapse, shift). Again, I wouldn't support that if it was in the evolution loop, but a few extra symmetry BCs in initial shouldn't really matter for speed unless its a very short run.

Also, keep in mind that I haven't actually put in any prints inside CartGrid3D and am going purely off of looking at the code. You may want to see if anything is actually being applied before changing the code or scheduling. I just thought it looked incorrect and thought you might want to check it out.

I had started a [ticket](https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/is...) a while back, so we can use that if there's any changes you want to make or questions for the maintainers.

Samuel Cupp    Postdoctoral Researcher    Department of Physics    University of Idaho

---

**From:** Miguel Zilhão <[mzilhao@ua.pt](mailto:mzilhao@ua.pt)> **Sent:** Saturday, January 21, 2023 3:05 AM **To:** Cupp, Samuel D. <[scupp1@my.apsu.edu](mailto:scupp1@my.apsu.edu)>; Helvi Witek <[hwitek@illinois.edu](mailto:hwitek@illinois.edu)> **Subject:** Re: LeanBSSNMoL scheduling

hi Sam,

thanks for your detailed reply. i understand better your point and i'm now a bit confused. let me elaborate below.

Thanks for the quick response. The ghost zones (inter-processor boundaries) wouldn't be triggered by this. They are triggered by

schedule LeanBSSN_adm2bssn at CCTK_INITIAL after ADMBase_PostInitial {     LANG: Fortran     OPTIONS: Local     SYNC: gammat } "Convert initial data into BSSN variables"

where the *SYNC* statement is.

indeed, you're totally right.

I asked Steve and Roland about when symmetry BCs are applied just to make sure I understood correctly, and the symmetry BCs are applied whenever the physical BCs are applied. However, physical BCs only get applied by the ApplyBCs function when a Select*ForBC function call has actually registered a function with the Boundary thorn. Since no Select*ForBC functions are scheduled before this ApplyBC call, it shouldn't do anything.

ah, wait. my understanding was that we were registering the symmetry BCs in the following call in the schedule file:

schedule LeanBSSN_symmetries at BASEGRID {    LANG: Fortran    OPTIONS: Global } "Register symmetries of the BSSN grid functions"

called at "BASEGRID". inside the routine "LeanBSSN_symmetries" we have

call SetCartSymVN( ierr, cctkGH, (/ 1, 1, 1/), "LeanBSSNMoL::conf_fac" )    call SetCartSymVN( ierr, cctkGH, (/ 1, 1, 1/), "LeanBSSNMoL::rhs_conf_fac" )

call SetCartSymVN( ierr, cctkGH, (/ 1, 1, 1/), "LeanBSSNMoL::hxx" )    call SetCartSymVN( ierr, cctkGH, (/-1,-1, 1/), "LeanBSSNMoL::hxy" )    call SetCartSymVN( ierr, cctkGH, (/-1, 1,-1/), "LeanBSSNMoL::hxz" )    call SetCartSymVN( ierr, cctkGH, (/ 1, 1, 1/), "LeanBSSNMoL::hyy" )    call SetCartSymVN( ierr, cctkGH, (/ 1,-1,-1/), "LeanBSSNMoL::hyz" )    call SetCartSymVN( ierr, cctkGH, (/ 1, 1, 1/), "LeanBSSNMoL::hzz" )

(...)

so what you're saying is that calling "ApplyBCs" after this is meaningless by itself unless one actually calls the Select*ForBC functions before? indeed, it seems we only call Select*ForBC much later (at MoL_PostStep).

Roland mentioned that "Correct. If there is not SelectBCs call then nothing will happen. It *is* possible that some of the old-style symmetry boundary conditions from CartGrid3D still get applied, which has some code in there (I think) to work with thorn Boundary."

it could be that there is something making sure that some symmetry conditions are applied, otherwise in every run with symmetry the gammat* grid functions would have non-initialized values at the symmetry points, which would likely create problems...

So I looked to see how CartGrid3D triggers its BCs, and it also relies on the selected variables from Boundary. So, best I can tell this does nothing. It isn't a real issue or anything, as its a few empty function calls that happens once at INITIAL. Removing it wouldn't make the code faster or anything, but I don't think it is needed. Alternatively, you may want a function SelectionVarForBC that actually causes gammat to have symmetry BCs applied.

i think this is probably the safest thing, just adding a line in the schedule file to make sure the "LeanBSSN_Boundaries" function is also called after at CCTK_INITIAL. what do you think?

thanks, Miguel

---

*From:* Miguel Zilhão <[mzilhao@ua.pt](mailto:mzilhao@ua.pt)> *Sent:* Friday, January 20, 2023 2:58 AM *To:* Helvi Witek <[hwitek@illinois.edu](mailto:hwitek@illinois.edu)>; Cupp, Samuel D. <[scupp1@my.apsu.edu](mailto:scupp1@my.apsu.edu)> *Subject:* Re: LeanBSSNMoL scheduling hi Sam and Helvi,

if i remember correctly, that call to "ApplyBCs" is there since in the routine LeanBSSN_adm2bssn we're not setting the values of the grid functions gammat{x,y,z} at the ghostzones (since this is done via some derivatives). we then call "ExtrapolateGammas" on these functions to set the values at the outer boundary, but the inter-processor and symmetry boundaries will remain unset.

with the call to "ApplyBCs", inter-processor and symmetry boundaries are applied, and the gammat* grid functions are then defined everywhere.

this was the reasoning, i think. does it make sense?

cheers, Miguel

On Thu, 2023-01-19 at 13:25 -0600, Helvi Witek wrote:

...

Dear Sam, let me also cc Miguel.

As far as I recall, this boundary scheduling is the same as for McLachlan. It is done right after the conversion from adm to bssn (including computing the conformal connection function).

@Miguel, do you recall the details of the "ApplyBCs" call here?

Best wishes, Helvi

---

Dr. Helvi Witek Assistant Professor Department of Physics University of Illinois at Urbana-Champaign

247 Loomis Lab 1110 W Green St Urbana, IL 61801

---

On 1/19/23 5:49 PM, Cupp, Samuel D. wrote:

...

Hey Helvi,     I'm wondering if maybe my previous email went to your junk. A while ago I had asked about LeanBSSNMoL. I'll just copy the previous email into this one.    I have been working on Baikal and BaikalVacuum (in wvuthorns), and they copied a lot of their scheduling from LeanBSSNMoL. In doing so, I noticed a weird scheduled group in the schedule.ccl that is also in LeanBSSNMoL. I was hoping you or someone else who worked on Lean could explain why it is there.    In LeanBSSNMoL's schedule.ccl is   schedule GROUP ApplyBCs as LeanBSSN_ApplyBCs at CCTK_INITIAL after LeanBSSN_ adm2bssn { } "Apply boundary conditions"    However, there are no boundary condition selection routines which run before this, and nothing is scheduled relative to this function. As best I can tell, this should do nothing and could just be removed. Is there something I'm missing? Or, should there be a SelectBCs routine running here that is missing? Thanks for your help.       Samuel Cupp     Postdoctoral Researcher     Department of Physics

‌

‌

-- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2648/empty-applybcs-cal...