hi all,
for the purposes of trying a particular gauge condition, we'd like to have a grid function (let's call it "shift_t0") that's set to be equal to the shift at t=0. we don't want this function to evolve at all; we just want this grid function to be accessible at all times (with exactly the same values).
is there something in particular that we need to watch out for, when doing this? we ask because we've noticed that when doing things in a "naive"* way, sometimes NaNs start developing at some point in the evolution (seemingly near refinement boundaries and when regriding). we noticed that exactly the same thing happens for the "puncture_u" grid function in TwoPunctures: this function is only used at t=0; however, if one activates storage for it and asks for its output, at some point NaNs start developing in the output, even though the function is never used again. so, what would be the standard way of defining a grid function that is constant in time?
thanks, Miguel & Helvi
* by "naive", we mean essentially the following
in schedule.ccl we have
STORAGE: shift_t0[1]
in interface.ccl we have
CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U" tags='prolongation="none"' { betax_t0 betay_t0 betaz_t0 } "shift vector at t=0"
and we set its value (on the whole grid) at a function call at CCTK_INITIAL after ADMBase_PostInitial
On 26 Oct 2017, at 23:23, Miguel Zilhão miguel.zilhao.nogueira@tecnico.ulisboa.pt wrote:
hi all,
for the purposes of trying a particular gauge condition, we'd like to have a grid function (let's call it "shift_t0") that's set to be equal to the shift at t=0. we don't want this function to evolve at all; we just want this grid function to be accessible at all times (with exactly the same values).
is there something in particular that we need to watch out for, when doing this? we ask because we've noticed that when doing things in a "naive"* way, sometimes NaNs start developing at some point in the evolution (seemingly near refinement boundaries and when regriding). we noticed that exactly the same thing happens for the "puncture_u" grid function in TwoPunctures: this function is only used at t=0; however, if one activates storage for it and asks for its output, at some point NaNs start developing in the output, even though the function is never used again. so, what would be the standard way of defining a grid function that is constant in time?
thanks, Miguel & Helvi
- by "naive", we mean essentially the following
in schedule.ccl we have
STORAGE: shift_t0[1]
in interface.ccl we have
CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U" tags='prolongation="none"' { betax_t0 betay_t0 betaz_t0 } "shift vector at t=0"
and we set its value (on the whole grid) at a function call at CCTK_INITIAL after ADMBase_PostInitial
Does this happen only after regridding? By not prolongating the refinement boundaries, it's not clear how the "new" bits of the grid are supposed to be filled when the fine grids move. Could this be the problem? If you give the gridfunction 3 timelevels and remove "prolongation=none", do the NaNs go away?
hi Ian,
for the purposes of trying a particular gauge condition, we'd like to have a grid function (let's call it "shift_t0") that's set to be equal to the shift at t=0. we don't want this function to evolve at all; we just want this grid function to be accessible at all times (with exactly the same values).
is there something in particular that we need to watch out for, when doing this? we ask because we've noticed that when doing things in a "naive"* way, sometimes NaNs start developing at some point in the evolution (seemingly near refinement boundaries and when regriding). we noticed that exactly the same thing happens for the "puncture_u" grid function in TwoPunctures: this function is only used at t=0; however, if one activates storage for it and asks for its output, at some point NaNs start developing in the output, even though the function is never used again. so, what would be the standard way of defining a grid function that is constant in time?
thanks, Miguel & Helvi
- by "naive", we mean essentially the following
in schedule.ccl we have
STORAGE: shift_t0[1]
in interface.ccl we have
CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U" tags='prolongation="none"' { betax_t0 betay_t0 betaz_t0 } "shift vector at t=0"
and we set its value (on the whole grid) at a function call at CCTK_INITIAL after ADMBase_PostInitial
Does this happen only after regridding? By not prolongating the refinement boundaries, it's not clear how the "new" bits of the grid are supposed to be filled when the fine grids move. Could this be the problem? If you give the gridfunction 3 timelevels and remove "prolongation=none", do the NaNs go away?
it seems to happen only after regridding, yes. we did try precisely that (using 3 timelevels and prolongating) but the NaNs still appeared...
thanks, Miguel
On 27 Oct 2017, at 10:28, Miguel Zilhão miguel.zilhao.nogueira@tecnico.ulisboa.pt wrote:
hi Ian,
for the purposes of trying a particular gauge condition, we'd like to have a grid function (let's call it "shift_t0") that's set to be equal to the shift at t=0. we don't want this function to evolve at all; we just want this grid function to be accessible at all times (with exactly the same values).
is there something in particular that we need to watch out for, when doing this? we ask because we've noticed that when doing things in a "naive"* way, sometimes NaNs start developing at some point in the evolution (seemingly near refinement boundaries and when regriding). we noticed that exactly the same thing happens for the "puncture_u" grid function in TwoPunctures: this function is only used at t=0; however, if one activates storage for it and asks for its output, at some point NaNs start developing in the output, even though the function is never used again. so, what would be the standard way of defining a grid function that is constant in time?
thanks, Miguel & Helvi
- by "naive", we mean essentially the following
in schedule.ccl we have
STORAGE: shift_t0[1]
in interface.ccl we have
CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U" tags='prolongation="none"' { betax_t0 betay_t0 betaz_t0 } "shift vector at t=0"
and we set its value (on the whole grid) at a function call at CCTK_INITIAL after ADMBase_PostInitial
Does this happen only after regridding? By not prolongating the refinement boundaries, it's not clear how the "new" bits of the grid are supposed to be filled when the fine grids move. Could this be the problem? If you give the gridfunction 3 timelevels and remove "prolongation=none", do the NaNs go away?
it seems to happen only after regridding, yes. we did try precisely that (using 3 timelevels and prolongating) but the NaNs still appeared...
The past timelevels need to be initialised. By default, Carpet will poison them with NaNs so that you can see when undefined values are used. Theoretically, if there is no time evolution of this variable, you should be able to use a single timelevel and use a tag 'prolongation="copy"', which should perform only spatial prolongation. Let me know what happens if you try this. I expect you will get an assertion failure from Carpet, and then you might want to try cherry-picking this commit:
cd repos/Carpet git cherry-pick d13a6e245a3d7d2b575094fd0def540d510a166c
which disables some code in Carpet that disables spatial prolongation in the case of "op_copy". The problem here likely resulted in confusion due to the operator being badly named (https://martinfowler.com/bliki/TwoHardThings.html). In arrangements/Carpet/CarpetLib/src/operators.hh, the implication is that the "copy" in op_copy refers to the time interpolation only, but the code in patched in the above commit assumes that "copy" means don't interpolate at all. It's not clear how this can make sense when the data comes from another refinement level, which will always have a different resolution, but maybe there is some case where this would have made sense.
Erik, should this commit be merged to master?
- by "naive", we mean essentially the following
in schedule.ccl we have
STORAGE: shift_t0[1]
in interface.ccl we have
CCTK_REAL shift_t0 type=gf timelevels=1 tags='tensortypealias="U" tags='prolongation="none"' { betax_t0 betay_t0 betaz_t0 } "shift vector at t=0"
and we set its value (on the whole grid) at a function call at CCTK_INITIAL after ADMBase_PostInitial
Does this happen only after regridding? By not prolongating the refinement boundaries, it's not clear how the "new" bits of the grid are supposed to be filled when the fine grids move. Could this be the problem? If you give the gridfunction 3 timelevels and remove "prolongation=none", do the NaNs go away?
it seems to happen only after regridding, yes. we did try precisely that (using 3 timelevels and prolongating) but the NaNs still appeared...
The past timelevels need to be initialised. By default, Carpet will poison them with NaNs so that you can see when undefined values are used. Theoretically, if there is no time evolution of this variable, you should be able to use a single timelevel and use a tag 'prolongation="copy"', which should perform only spatial prolongation. Let me know what happens if you try this. I expect you will get an assertion failure from Carpet,
indeed, this is what happens. i got:
WARNING level 0 from host meurglysIII process 0 while executing schedule bin (none), routine (no thorn)::(no routine) in thorn CarpetLib, file /home/mzilhao/dev/ET/Cactus/arrangements/Carpet/CarpetLib/src/data.cc:615: -> There is no vertex-centred stencil for op="LAGRANGE" or op="COPY" with order_space not in {1, 3, 5, 7, 9, 11} -------------------------------------------------------------------------- mpirun noticed that process rank 1 with PID 11602 on node meurglysIII exited on signal 6 (Aborted). --------------------------------------------------------------------------
and then you might want to try cherry-picking this commit:
cd repos/Carpet git cherry-pick d13a6e245a3d7d2b575094fd0def540d510a166c
which disables some code in Carpet that disables spatial prolongation in the case of "op_copy". The problem here likely resulted in confusion due to the operator being badly named (https://martinfowler.com/bliki/TwoHardThings.html). In arrangements/Carpet/CarpetLib/src/operators.hh, the implication is that the "copy" in op_copy refers to the time interpolation only, but the code in patched in the above commit assumes that "copy" means don't interpolate at all. It's not clear how this can make sense when the data comes from another refinement level, which will always have a different resolution, but maybe there is some case where this would have made sense.
Erik, should this commit be merged to master?
with this commit it seems to be working fine, yes! unless there is some other way of doing this, it would be great if this commit could be merged to master.
many thanks, Miguel
Hello Miguel, all,
with this commit it seems to be working fine, yes! unless there is some other way of doing this, it would be great if this commit could be merged to master.
I think there is one more roadblock to consider: when you have 3 timelevels then Carpet will cycle the timelevels after each step, shuffling shift0 into shift0_p and shift0_p into shift0_pp and (at least making it look like it was) allocating fresh memory for shift0. If you have enabled poison_new_timelevels (off by default I think) then you'd see poison in the new timelevel. This will happen whenever you have 3 timelevels.
The solution would be to have only a single timelevel and do eg what CartGrid3D does and schedule your setting routine in BASEGRID, POSTREGRIDINITIAL and POSTREGRID:
schedule CartGrid3D_SetCoordinates as SpatialCoordinates at CCTK_BASEGRID { LANG:C WRITES: grid::coordinates(Everywhere) } "Set up spatial 3D Cartesian coordinates on the GH"
schedule CartGrid3D_SetCoordinates as SpatialCoordinates at CCTK_POSTREGRIDINITIAL { LANG: C WRITES: grid::coordinates(Everywhere) } "Set Coordinates after regridding"
schedule CartGrid3D_SetCoordinates as SpatialCoordinates at CCTK_POSTREGRID { LANG: C WRITES: grid::coordinates(Everywhere) } "Set Coordinates after regridding"
you can also *try* to see if removing the "prolongation_opertaor='none'" setting in you interface.ccl *and* also making sure to regrid only during coarse steps (so that interpolation in time is not needed).
Yours, Roland
hi Roland and all,
we briefly tested this, but it didn't seem to be of much help... for our purposes having just one timelevel should suffice in any case, so the fix by Ian would be more convenient, if possible.
many thanks, Miguel
On 27/10/17 12:48, Roland Haas wrote:
Hello Miguel, all,
with this commit it seems to be working fine, yes! unless there is some other way of doing this, it would be great if this commit could be merged to master.
I think there is one more roadblock to consider: when you have 3 timelevels then Carpet will cycle the timelevels after each step, shuffling shift0 into shift0_p and shift0_p into shift0_pp and (at least making it look like it was) allocating fresh memory for shift0. If you have enabled poison_new_timelevels (off by default I think) then you'd see poison in the new timelevel. This will happen whenever you have 3 timelevels.
The solution would be to have only a single timelevel and do eg what CartGrid3D does and schedule your setting routine in BASEGRID, POSTREGRIDINITIAL and POSTREGRID:
schedule CartGrid3D_SetCoordinates as SpatialCoordinates at CCTK_BASEGRID { LANG:C WRITES: grid::coordinates(Everywhere) } "Set up spatial 3D Cartesian coordinates on the GH"
schedule CartGrid3D_SetCoordinates as SpatialCoordinates at CCTK_POSTREGRIDINITIAL { LANG: C WRITES: grid::coordinates(Everywhere) } "Set Coordinates after regridding"
schedule CartGrid3D_SetCoordinates as SpatialCoordinates at CCTK_POSTREGRID { LANG: C WRITES: grid::coordinates(Everywhere) } "Set Coordinates after regridding"
you can also *try* to see if removing the "prolongation_opertaor='none'" setting in you interface.ccl *and* also making sure to regrid only during coarse steps (so that interpolation in time is not needed).
Yours, Roland
users@lists.einsteintoolkit.org
-
Ian Hinder -
Miguel Zilhão -
Roland Haas