Meeting minutes:
Present were: Tanja, Roland, Ian, Eloisa, Erik, Bruno, Yosef
ET paper: * Bruno still has troubles with RotatingSymmetry180 when using the development version of ET for the BBH example (parfile is in subversion) * will try running without RotatingSymmetry180 * convergence in norms is bad, somewhat better in 1D data for a short time after simulation startup * cosmology example is done for now * other examples had no maintainer on the phone call * example data: ** add parfiles to paper repository ** add data for plots to paper repository ** have single person generate all plots for final version to achieve uniform style
Array padding: * Erik suggests introducing array padding into Cactus to avoid cache aliasing issues * re-use existing but (essentially) unused cctk_lssh for loop boundaries of i,j,k * would use cctk_lsh to compute 1D index from i,j,k * would loop i,j,k using cctk_lssh * Cactus doc's already say to use cctk_lssh for this * thorns that are not updated *might* only compute extra unused data for the "padding" points ** thorns using ghostzones/boundary zones as cctk_lsh-nghostzones/boundary_width might cause problems ** thorns computing in the interior might compute extra data and overwrite ghostzone/boundary information * would like to have a mechanism to check for errors during runtime * make padding a runtime parameter in the parameter files * provide example parfile demonstrating speedup
Yours, Roland
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
Cheers, Bruno.
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Time prolongation is second order accurate.
You can use tapered grids, which avoids all time interpolation except possibly during regridding. If you are careful about regridding you don't need time interpolation for this either. This gives you clean fourth order convergence, except near the outer boundary if you are cheating there (and we all are).
Another issue to consider is how you set up the past timelevels. If you copy the data from the current time level, then you are introducing a first order error. If you use the three-timelevel-initialisation, then you are second order accurate. Again, these past timelevels are used only for time interpolation, but they can reduce the convergence order even further if they are not well initialised.
-erik
On 24 May 2011, at 17:50, Erik Schnetter wrote:
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Time prolongation is second order accurate.
As I understand it, the three-point interpolation (from the three timelevels) is locally 3rd order accurate, but when you add up T/dt of them to get to a fixed time, you reduce the order to 2. I tend to get confused about this, but do we agree on this?
On Tue, May 24, 2011 at 12:00 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 24 May 2011, at 17:50, Erik Schnetter wrote:
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Time prolongation is second order accurate.
As I understand it, the three-point interpolation (from the three timelevels) is locally 3rd order accurate, but when you add up T/dt of them to get to a fixed time, you reduce the order to 2. I tend to get confused about this, but do we agree on this?
Counting orders is always difficult. There are three time levels for each point, which prescribe a parabola. This is used to interpolate, with an error term in O(dt^3). If you add up T/dt to get to a fixed time, the error is in O(dt^2), which I would call "first order accurate".
-erik
Hi Erik and Ian,
Erik Schnetter wrote:
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Thanks! I missed that...
Time prolongation is second order accurate.
Right. We use three time levels (equivalently three points) to prolongate, so it should be second order, O(dt^2), accurate.
You can use tapered grids, which avoids all time interpolation except possibly during regridding. If you are careful about regridding you don't need time interpolation for this either. This gives you clean fourth order convergence, except near the outer boundary if you are cheating there (and we all are).
What do you mean by "If you are careful about regridding you don't need time interpolation for this either. "? You mean besides using tapered grids, only regrid when all grids are aligned (in time), ie in only at the coarsest time steps?
Another issue to consider is how you set up the past timelevels. If you copy the data from the current time level, then you are introducing a first order error.
Good point. I guess that's how it is set right now: Carpet::init_fill_timelevels = "yes" InitBase::initial_data_setup_method = "init_all_levels"
If you use the
three-timelevel-initialisation, then you are second order accurate. Again, these past timelevels are used only for time interpolation, but they can reduce the convergence order even further if they are not well initialised.
But to use init_3_timelevels we need to have all time levels on the ratio 2:1 (for example). We can't have as it is set there right now:
Carpet::time_refinement_factors = "[1, 1, 2, 4,...]"
Cheers, Bruno.
-erik
On Tue, May 24, 2011 at 11:43 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Hi Erik and Ian,
Erik Schnetter wrote:
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Thanks! I missed that...
Time prolongation is second order accurate.
Right. We use three time levels (equivalently three points) to prolongate, so it should be second order, O(dt^2), accurate.
Yes, it uses three points. Not everybody counts orders the same way I did here.
You can use tapered grids, which avoids all time interpolation except possibly during regridding. If you are careful about regridding you don't need time interpolation for this either. This gives you clean fourth order convergence, except near the outer boundary if you are cheating there (and we all are).
What do you mean by "If you are careful about regridding you don't need time interpolation for this either. "? You mean besides using tapered grids, only regrid when all grids are aligned (in time), ie in only at the coarsest time steps?
Not quite. All the grids that are changing need to be aligned with their next coarser grids. If e.g. the first three levels don't change, then you can regrid much more often than just at full coarse grid time steps.
Another issue to consider is how you set up the past timelevels. If you copy the data from the current time level, then you are introducing a first order error.
Good point. I guess that's how it is set right now: Carpet::init_fill_timelevels = "yes" InitBase::initial_data_setup_method = "init_all_levels"
Yes, this will introduce a very large error during time "interpolation", because by doing so, you essentially copy the initial data to the future times instead of interpolating.
If you use the
three-timelevel-initialisation, then you are second order accurate. Again, these past timelevels are used only for time interpolation, but they can reduce the convergence order even further if they are not well initialised.
But to use init_3_timelevels we need to have all time levels on the ratio 2:1 (for example). We can't have as it is set there right now:
Carpet::time_refinement_factors = "[1, 1, 2, 4,...]"
Yes. Sorry.
-erik
Hi Erik:
Erik Schnetter wrote:
On Tue, May 24, 2011 at 11:43 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Hi Erik and Ian,
Erik Schnetter wrote:
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
- Bruno still has troubles with RotatingSymmetry180 when using the
development version of ET for the BBH example (parfile is in subversion)
- will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
- convergence in norms is bad, somewhat better in 1D data for a short
time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Thanks! I missed that...
Time prolongation is second order accurate.
Right. We use three time levels (equivalently three points) to prolongate, so it should be second order, O(dt^2), accurate.
Yes, it uses three points. Not everybody counts orders the same way I did here.
You can use tapered grids, which avoids all time interpolation except possibly during regridding. If you are careful about regridding you don't need time interpolation for this either. This gives you clean fourth order convergence, except near the outer boundary if you are cheating there (and we all are).
What do you mean by "If you are careful about regridding you don't need time interpolation for this either. "? You mean besides using tapered grids, only regrid when all grids are aligned (in time), ie in only at the coarsest time steps?
Not quite. All the grids that are changing need to be aligned with their next coarser grids. If e.g. the first three levels don't change, then you can regrid much more often than just at full coarse grid time steps.
Then the following parameters should help achieving clean 4th order convergence:
Carpet::use_tapered_grids = "yes" CarpetRegrid2::freeze_unaligned_levels = "yes"
and we would end up with a buffer zone = 2 * 4 * 3 = 24 points (time refinement ratio * # of RK4 algorithmic steps * width of 4th order centered dissipation stencil) plus the number of ghost zones around each grid component (3 in this case). Is this counting correct? this seems quite expensive...
Another issue to consider is how you set up the past timelevels. If you copy the data from the current time level, then you are introducing a first order error.
Good point. I guess that's how it is set right now: Carpet::init_fill_timelevels = "yes" InitBase::initial_data_setup_method = "init_all_levels"
Yes, this will introduce a very large error during time "interpolation", because by doing so, you essentially copy the initial data to the future times instead of interpolating.
yes, but this first order error would be evident only on the buffer zones, right? and I would expect this error to pollute the solution later on in the simulation only, not on the first few time steps.
Cheers, Bruno.
If you use the
three-timelevel-initialisation, then you are second order accurate. Again, these past timelevels are used only for time interpolation, but they can reduce the convergence order even further if they are not well initialised.
But to use init_3_timelevels we need to have all time levels on the ratio 2:1 (for example). We can't have as it is set there right now:
Carpet::time_refinement_factors = "[1, 1, 2, 4,...]"
Yes. Sorry.
-erik
On Wed, May 25, 2011 at 12:56 AM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Hi Erik:
Erik Schnetter wrote:
On Tue, May 24, 2011 at 11:43 PM, Bruno C. Mundim bcmsma@astro.rit.edu wrote:
Hi Erik and Ian,
Erik Schnetter wrote:
On Tue, May 24, 2011 at 2:11 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 23 May 2011, at 23:49, Bruno Coutinho Mundim wrote:
> * Bruno still has troubles with RotatingSymmetry180 when using the > development version of ET for the BBH example (parfile is in > subversion) > * will try running without RotatingSymmetry180
Just got the results: no problem without RotatingSymmetry180.
OK good, so we know where the problem is.
> * convergence in norms is bad, somewhat better in 1D data for a short > time after simulation startup
A closer look into the initial data revealed that both the l2-norm of the hamiltonian constraint and its value along the x-axis converge to the expected order, 4th order. This convergence is not observed anymore in the very next coarse step when the comparison is done again.
The time prolongation is only 3rd order accurate so I wouldn't expect convergence at 4th order.
Thanks! I missed that...
Time prolongation is second order accurate.
Right. We use three time levels (equivalently three points) to prolongate, so it should be second order, O(dt^2), accurate.
Yes, it uses three points. Not everybody counts orders the same way I did here.
You can use tapered grids, which avoids all time interpolation except possibly during regridding. If you are careful about regridding you don't need time interpolation for this either. This gives you clean fourth order convergence, except near the outer boundary if you are cheating there (and we all are).
What do you mean by "If you are careful about regridding you don't need time interpolation for this either. "? You mean besides using tapered grids, only regrid when all grids are aligned (in time), ie in only at the coarsest time steps?
Not quite. All the grids that are changing need to be aligned with their next coarser grids. If e.g. the first three levels don't change, then you can regrid much more often than just at full coarse grid time steps.
Then the following parameters should help achieving clean 4th order convergence:
Carpet::use_tapered_grids = "yes" CarpetRegrid2::freeze_unaligned_levels = "yes"
and we would end up with a buffer zone = 2 * 4 * 3 = 24 points (time refinement ratio * # of RK4 algorithmic steps * width of 4th order centered dissipation stencil) plus the number of ghost zones around each grid component (3 in this case). Is this counting correct? this seems quite expensive...
Yes to all the above.
Another issue to consider is how you set up the past timelevels. If you copy the data from the current time level, then you are introducing a first order error.
Good point. I guess that's how it is set right now: Carpet::init_fill_timelevels = "yes" InitBase::initial_data_setup_method = "init_all_levels"
Yes, this will introduce a very large error during time "interpolation", because by doing so, you essentially copy the initial data to the future times instead of interpolating.
yes, but this first order error would be evident only on the buffer zones, right? and I would expect this error to pollute the solution later on in the simulation only, not on the first few time steps.
Yes, it should affect only the buffer zones. However, given that the solution travels 24 grid points per time step (as you just calculated above), the prolongation error can well affect the interior of the fine grids quite quickly, in particular a large error introduced by wrong past time levels.
In principle, our BBH initial data are in equilibrium, and if one chose a co-rotating frame (that maybe also has a bit of infall), then the past time levels would be much closer to being correct. That is, if one rotated (and pushed slightly outwards) the past time levels, the error in the past timelevels should be greatly reduced. All that is needed for this is a coordinate transformation, which should be computable from the corotation angular velocity (and the initial radial momenta).
-erik
If you use the
three-timelevel-initialisation, then you are second order accurate. Again, these past timelevels are used only for time interpolation, but they can reduce the convergence order even further if they are not well initialised.
But to use init_3_timelevels we need to have all time levels on the ratio 2:1 (for example). We can't have as it is set there right now:
Carpet::time_refinement_factors = "[1, 1, 2, 4,...]"
Yes. Sorry.
-erik
On Mon, May 23, 2011 at 11:48 AM, Roland Haas roland.haas@physics.gatech.edu wrote:
Array padding:
- Erik suggests introducing array padding into Cactus to avoid cache
aliasing issues
- re-use existing but (essentially) unused cctk_lssh for loop boundaries of
i,j,k
- would use cctk_lsh to compute 1D index from i,j,k
- would loop i,j,k using cctk_lssh
- Cactus doc's already say to use cctk_lssh for this
- thorns that are not updated *might* only compute extra unused data for the
"padding" points ** thorns using ghostzones/boundary zones as cctk_lsh-nghostzones/boundary_width might cause problems ** thorns computing in the interior might compute extra data and overwrite ghostzone/boundary information
- would like to have a mechanism to check for errors during runtime
- make padding a runtime parameter in the parameter files
- provide example parfile demonstrating speedup
Regarding array padding, I just found this page http://igoro.com/archive/gallery-of-processor-cache-effects/. Example 5 there documents the effects of padding. If we access (or update) every 512th byte of an array, then this is significantly slower than updating e.g. every 520th byte. This corresponds to making a 2D array slightly larger, so that points in the j direction are separated not by 512, but by 520 bytes.
-erik
users@lists.einsteintoolkit.org
-
Bruno C. Mundim -
Bruno Coutinho Mundim
-
Erik Schnetter -
Ian Hinder -
Roland Haas