Dear all,
as already mentioned in the meeting minutes and discussed in last week's CIGR call, Christian Reisswig has been working on adding Llama Multipatch compatibility to GRHydro. For those of you not familiar with Llama, this is the description from the Llama website (www.llamacode.org): --8<-- The Llama code is a 3-dimensional multiblock infrastructure with adaptive mesh-refinement for Cactus based on Carpet. It provides different patch systems that cover the simulation domain by a set of overlapping patches. Each of these patches has local cooordinates with a well defined relation to global Cartesian coordinates. However, all computations are carried out using a global Cartesian tensor basis such that involved tensor transformations between patch systems can be avoided. Information between the different patches is communicated via interpolation in the overlap zones. --8<--
We would like to incorporate these changes into the public development version of GRHydro. Right now multipatch support is functional for pure hydro simulations but still incomplete (eg. only the HLLE solver is supported, others will currently fail with multipatch enabled). No (serious) attempt has been made to incorporate MP into MHD, though the changes required should be straightforward. The proposed patch is backward compatible in that old parameter files will continue to work without change.
However the patch touches a large number of code lines since all references to vector/tensor components have to be replaced with references to tensor components in a local tensor basis. In practice this means that all occurances of gxx, betax, vel[i] etc. have to be replaced by gaa, betaa, lvlel[i]. On top of that, the requirement to stay backwards compatible (and to not use extra memory) with old parameter files when MP is not actually employed adds another layer so that in the end gxx is replaced by g11 etc. This is all rather straightforward, but unfortunately tedious and makes for a large patch.
We have tested that the code runs the old GRHydro testsuites and passes the same testsuites that a current vanilla checkout passes.
We will begin adding multipatch support during the day.
Yours, Christian R., Christian O. and Roland
On 15 Sep 2011, at 17:10, Roland Haas wrote:
Dear all,
as already mentioned in the meeting minutes and discussed in last week's CIGR call, Christian Reisswig has been working on adding Llama Multipatch compatibility to GRHydro. For those of you not familiar with Llama, this is the description from the Llama website (www.llamacode.org): --8<-- The Llama code is a 3-dimensional multiblock infrastructure with adaptive mesh-refinement for Cactus based on Carpet. It provides different patch systems that cover the simulation domain by a set of overlapping patches. Each of these patches has local cooordinates with a well defined relation to global Cartesian coordinates. However, all computations are carried out using a global Cartesian tensor basis such that involved tensor transformations between patch systems can be avoided. Information between the different patches is communicated via interpolation in the overlap zones. --8<--
We would like to incorporate these changes into the public development version of GRHydro. Right now multipatch support is functional for pure hydro simulations but still incomplete (eg. only the HLLE solver is supported, others will currently fail with multipatch enabled). No (serious) attempt has been made to incorporate MP into MHD, though the changes required should be straightforward. The proposed patch is backward compatible in that old parameter files will continue to work without change.
However the patch touches a large number of code lines since all references to vector/tensor components have to be replaced with references to tensor components in a local tensor basis. In practice this means that all occurances of gxx, betax, vel[i] etc. have to be replaced by gaa, betaa, lvlel[i]. On top of that, the requirement to stay backwards compatible (and to not use extra memory) with old parameter files when MP is not actually employed adds another layer so that in the end gxx is replaced by g11 etc. This is all rather straightforward, but unfortunately tedious and makes for a large patch.
We have tested that the code runs the old GRHydro testsuites and passes the same testsuites that a current vanilla checkout passes.
We will begin adding multipatch support during the day.
Do you know if there is any performance penalty when multipatch is switched off?
Hello Ian, all,
Do you know if there is any performance penalty when multipatch is switched off?
I did not do extensive testing, but for the simple TOV only test that I did, I found no measurable slowdown when multipatch was switchted off compared not compiling in Multipatch.
I ran the TOVSolver testsuite test_tov_carpet.par with HLLE solver and TVD reconstruction and 64^3 grid for 100 iterations. The differences between using pointers and straight arrays was about 1%, sometimes (when running the same run more than once) pointers were faster, sometimes arrays (this is for gcc 4.6, on my laptop on a single core).
If you look at what was done then essentially pointers were introduced which point either to the local tensor variables (gaa) or to the global ones (gxx). Apparently using pointers still lets the compiler do all the optimizations and keeps track of aliasing. I do not know if this might change once we use cctk_lsh in the flesh to declare array dimensions rather than passing three individual integers per grid function.
So, no to my knowledge there is no performance penalty. Things will likely be different if one actually were doing the transformations but with a trivial jacobian (which would make the code a bit shorter and easier to read in some parts in part. GRHydro_Transform*.c).
Yours, Roland
On 15 Sep 2011, at 17:10, Roland Haas wrote:
Dear all,
as already mentioned in the meeting minutes and discussed in last week's CIGR call, Christian Reisswig has been working on adding Llama Multipatch compatibility to GRHydro. For those of you not familiar with
...
We have tested that the code runs the old GRHydro testsuites and passes the same testsuites that a current vanilla checkout passes.
We will begin adding multipatch support during the day.
The tests for HydroInitExcision started failing with NaNs on 17-Sep-2011.
http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/
The diff between the source tree that passed and the source tree that failed is at
http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/einsteintoolki...
and it seems to contain the multipatch work. Could you check again if the test passes for you with the current version?
users@lists.einsteintoolkit.org
-
Ian Hinder -
Roland Haas