Hi Erik,
Frank suggested you might be an appropriate person to ask about elliptic solver thorns that work in the current release of ET. I have, as you know, built Hilbert and have built the development version.
I am wanting to write a thorn which prepares a geon, approximated by initial data which has toroidal symmetry and is bounded. The desired distribution would be placed on the r axis in cylindrical coordinates and have a gaussian distribution. In the actual application, of course the initial data would be interpolated onto a Cartesian coordinate mesh or meshes. I just want to solve the Hamiltonian constraint and have the initial data be time-symmetric. This sounds like a pretty simple thing to do. I think I could start with Brill and put q to my purposes and pretty much get out what I want. However, the IDBrill does not seem to work. Also, I am wondering what kinds of elliptic solvers you know of that are fully supported in ET? Frank mentioned a petsc thorn. It does not get downloaded and built by default and looking at the einsteintoolkit.th file it is mentioned in:
# CactusElliptic thorns !TARGET = $ARR !TYPE = git !URL = https://bitbucket.org/cactuscode/cactuselliptic.git !REPO_PATH= $2 !CHECKOUT = CactusElliptic/EllPETSc CactusElliptic/TATPETSc CactusElliptic/EllBase #DISABLED CactusElliptic/EllPETSc CactusElliptic/EllSOR CactusElliptic/TATelliptic #DISABLED CactusElliptic/TATPETSc
However, what's with the DISABLED reference to EllPETSc is a bit concerning. So before I go though the exercise to look more into the EllPETSc, I thought I'd ask whether the EllPETSc thorn is fully compatible with Hilbert?
Can you thus please bring me up to the current moment on just what elliptic solvers are in use for initial data construction in Hilbert? I know many people have been focussing on the multiple BH problem, so there's not much traffic anymore about Brill. I do want to run Brill though and also do the "geon" problem.
Thanks for your advice and any pointers you can give.
Comer
On 27 May 2015, at 20:50, Comer Duncan comer.duncan@gmail.com wrote:
Hi Erik,
Frank suggested you might be an appropriate person to ask about elliptic solver thorns that work in the current release of ET. I have, as you know, built Hilbert and have built the development version.
Hi Comer,
There is a page (https://docs.einsteintoolkit.org/et-docs/Generic_elliptic_solver) on the wiki describing our attempts to round up all the elliptic solvers in November 2011. I don't think it's linked from https://docs.einsteintoolkit.org, but I found it with a search.
Since this list was written, a multigrid solver has been developed by Eloisa Bentivegna, and has been proposed for inclusion in the ET. The corresponding ticket is https://trac.einsteintoolkit.org/ticket/1676. It is a part of the Cosmo Toolkit (https://bitbucket.org/eloisa/cosmology), a collection of Cactus thorns focused on cosmology.
I include below the email Eloisa sent to the ET list on 28-Aug-2013. We are going to push to get this included in the toolkit as soon as possible, but definitely before the next release. According to the ticket, we need to provide some examples for the gallery page before it can be included, and this is what is holding it up!
Regarding the last part of your question,
Can you thus please bring me up to the current moment on just what elliptic solvers are in use for initial data construction in Hilbert? I know many people have been focussing on the multiple BH problem, so there's not much traffic anymore about Brill. I do want to run Brill though and also do the "geon" problem.
I think you have characterised the situation quite well; most focus has been on TwoPuncture for binary black holes, and LORENE for binary neutron stars.
As far as I can tell, the IDBrillData thorn can use either BAM_elliptic, SOR or petsc, as you can see from its param.ccl. I have never used this thorn, but my guess is that it can be made to work. In fact, in the par directory is an example using SOR, from the CactusElliptic arrangement, which IS in the ET:
IDBrillData/par/brilldata.par
You could also give that a try, if you are willing to lower yourself to using SOR :) I just ran it, and it spits out some warnings about symmetry boundary conditions, but it generates a metric which isn't 0 or NaN, so there's a good chance it works!
sim create-run arrangements/EinsteinInitialData/IDBrillData/par/brilldata.par
Here is Eloisa's email about the multigrid solver:
On 28 Aug 2013, at 17:56, Eloisa Bentivegna bentivegna@cct.lsu.edu wrote:
Dear Cactus and Einstein Toolkit users,
I have been working on a solver for elliptic PDEs based on the multigrid paradigm, and developed as a Cactus thorn using Carpet. The solver is now able to tackle some basic problems and could be applied to other cases of interest to this community. You can obtain a copy by checking out the Cosmology arrangement from its repository:
git clone git@bitbucket.org:eloisa/cosmology.git
There you will also find documentation, license information, and a test suite. There is no tutorial, but I think that what is under docs/ and par/ should suffice to get most people started. For more information, you can also peruse http://arxiv.org/abs/arXiv:1305.5576, especially if you are interested in elliptic problems on spaces without boundaries.
Please notice that the spirit of the solver is not to replace existing state-of-the-art tools, but to provide a way of solving elliptic PDEs with is both easy to learn and to extend, and gives reasonable answers in a reasonable amount of time. In order to keep this flexibility, the solver is simply unable to compete with other software tailored to specific equations or geometries. In areas where these alternatives exist, they should be used instead.
Finally, beware of the lack of explicit support for some of the latest Carpet features, such as multipatch and cell-centered grids. Being unfamiliar with these features, I cannot estimate how much work would be necessary to include them.
I hope many of you will find this tool useful, and I look forward to hear your feedback.
Eloisa
Users mailing list Users@cactuscode.org http://cactuscode.org/mailman/listinfo/users
On Wed, May 27, 2015 at 2:50 PM, Comer Duncan comer.duncan@gmail.com wrote:
Hi Erik,
Frank suggested you might be an appropriate person to ask about elliptic solver thorns that work in the current release of ET. I have, as you know, built Hilbert and have built the development version.
I am wanting to write a thorn which prepares a geon, approximated by initial data which has toroidal symmetry and is bounded. The desired distribution would be placed on the r axis in cylindrical coordinates and have a gaussian distribution. In the actual application, of course the initial data would be interpolated onto a Cartesian coordinate mesh or meshes. I just want to solve the Hamiltonian constraint and have the initial data be time-symmetric. This sounds like a pretty simple thing to do. I think I could start with Brill and put q to my purposes and pretty much get out what I want. However, the IDBrill does not seem to work. Also, I am wondering what kinds of elliptic solvers you know of that are fully supported in ET? Frank mentioned a petsc thorn. It does not get downloaded and built by default and looking at the einsteintoolkit.th file it is mentioned in:
# CactusElliptic thorns !TARGET = $ARR !TYPE = git !URL = https://bitbucket.org/cactuscode/cactuselliptic.git !REPO_PATH= $2 !CHECKOUT = CactusElliptic/EllPETSc CactusElliptic/TATPETSc CactusElliptic/EllBase #DISABLED CactusElliptic/EllPETSc CactusElliptic/EllSOR CactusElliptic/TATelliptic #DISABLED CactusElliptic/TATPETSc
However, what's with the DISABLED reference to EllPETSc is a bit concerning. So before I go though the exercise to look more into the EllPETSc, I thought I'd ask whether the EllPETSc thorn is fully compatible with Hilbert?
Can you thus please bring me up to the current moment on just what elliptic solvers are in use for initial data construction in Hilbert? I know many people have been focussing on the multiple BH problem, so there's not much traffic anymore about Brill. I do want to run Brill though and also do the "geon" problem.
Thanks for your advice and any pointers you can give.
Comer
The #DISABLED just means that it has been disabled, not that the thorn is bad. We only disable it because PETSc is a large package and may be difficult to install, and only very few people use PETSc with Cactus. I am regularly building with PETSc, and have also run the ET tests with PETSc enable on some of our machines.
I don't recommend using SOR. If you use mesh refinement, or if you use a machine more powerful than a laptop, then you can afford domains that are so large that SOR converges too slowly to be useful.
BAM_Elliptic is not freely available. It also has not been used in more than a decade, and I doubt it would be useful to resurrect it.
PETSc is a modern, full-featured elliptic solver. Unfortunately, there is no integration with Cactus's grid functions if you want to use mesh refinement. If you can live with a unigrid simulation, then EllPETSc allows you to solve linear elliptic equations. The thorns EllBase or EllPETSc should have some documentation on this; if not, maybe we can discuss next Monday during the ET telecon.
There is also a thorn TATPETSc that can solve non-linear elliptic equations (also unigrid only).
I haven't used these thorns in production in quite some time. Our initial conditions typically come from third-party solvers, e.g. Lorene or TwoPunctures.
The most modern solution is probably the multigrid solver that Eloisa wrote and that Ian described.
The situation of elliptic solvers in Cactus is not as good as I would like it to be. To help you out, it is probably best to talk in person, so that we see what you need. Do you have time next Monday to call in to our ET telecon?
-erik
Hi Erik,
Thanks for your informative take on the status of elliptic solvers. I am out on travel at the moment and will be traveling home next Monday. So maybe the next next Monday?
From what you describe it seems clear that having an elliptic solver which
knows about mesh refinement is sort of hard. Is the multigrid solver of Eloisa Carpet aware? If so, I would probably elect to use it so long as it is designed to work for asymtotically flat initial data.
If not, I wonder whether it would be reasonable to take the output of a solver done on unigrid and read it and remap it onto an AMR grid? Then run it as usual in ET. Either make the solver itself internally a Carpet aware thing or try to do it after the solver is done. Not an ideal way perhaps but perhaps a workaround.
Comer
On Thu, May 28, 2015 at 11:30 PM, Erik Schnetter schnetter@cct.lsu.edu wrote:
On Wed, May 27, 2015 at 2:50 PM, Comer Duncan comer.duncan@gmail.com wrote:
Hi Erik,
Frank suggested you might be an appropriate person to ask about elliptic solver thorns that work in the current release of ET. I have, as you know, built Hilbert and have built the development version.
I am wanting to write a thorn which prepares a geon, approximated by initial data which has toroidal symmetry and is bounded. The desired distribution would be placed on the r axis in cylindrical coordinates and have a gaussian distribution. In the actual application, of course the initial data would be interpolated onto a Cartesian coordinate mesh or meshes. I just want to solve the Hamiltonian constraint and have the initial data be time-symmetric. This sounds like a pretty simple thing to do. I think I could start with Brill and put q to my purposes and pretty much get out what I want. However, the IDBrill does not seem to work. Also, I am wondering what kinds of elliptic solvers you know of that are fully supported in ET? Frank mentioned a petsc thorn. It does not get downloaded and built by default and looking at the einsteintoolkit.th file it is mentioned in:
# CactusElliptic thorns !TARGET = $ARR !TYPE = git !URL = https://bitbucket.org/cactuscode/cactuselliptic.git !REPO_PATH= $2 !CHECKOUT = CactusElliptic/EllPETSc CactusElliptic/TATPETSc CactusElliptic/EllBase #DISABLED CactusElliptic/EllPETSc CactusElliptic/EllSOR CactusElliptic/TATelliptic #DISABLED CactusElliptic/TATPETSc
However, what's with the DISABLED reference to EllPETSc is a bit concerning. So before I go though the exercise to look more into the EllPETSc, I thought I'd ask whether the EllPETSc thorn is fully compatible with Hilbert?
Can you thus please bring me up to the current moment on just what elliptic solvers are in use for initial data construction in Hilbert? I know many people have been focussing on the multiple BH problem, so there's not much traffic anymore about Brill. I do want to run Brill though and also do the "geon" problem.
Thanks for your advice and any pointers you can give.
Comer
The #DISABLED just means that it has been disabled, not that the thorn is bad. We only disable it because PETSc is a large package and may be difficult to install, and only very few people use PETSc with Cactus. I am regularly building with PETSc, and have also run the ET tests with PETSc enable on some of our machines.
I don't recommend using SOR. If you use mesh refinement, or if you use a machine more powerful than a laptop, then you can afford domains that are so large that SOR converges too slowly to be useful.
BAM_Elliptic is not freely available. It also has not been used in more than a decade, and I doubt it would be useful to resurrect it.
PETSc is a modern, full-featured elliptic solver. Unfortunately, there is no integration with Cactus's grid functions if you want to use mesh refinement. If you can live with a unigrid simulation, then EllPETSc allows you to solve linear elliptic equations. The thorns EllBase or EllPETSc should have some documentation on this; if not, maybe we can discuss next Monday during the ET telecon.
There is also a thorn TATPETSc that can solve non-linear elliptic equations (also unigrid only).
I haven't used these thorns in production in quite some time. Our initial conditions typically come from third-party solvers, e.g. Lorene or TwoPunctures.
The most modern solution is probably the multigrid solver that Eloisa wrote and that Ian described.
The situation of elliptic solvers in Cactus is not as good as I would like it to be. To help you out, it is probably best to talk in person, so that we see what you need. Do you have time next Monday to call in to our ET telecon?
-erik
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
On Fri, May 29, 2015 at 10:40 AM, Comer Duncan comer.duncan@gmail.com wrote:
Hi Erik,
Thanks for your informative take on the status of elliptic solvers. I am out on travel at the moment and will be traveling home next Monday. So maybe the next next Monday?
From what you describe it seems clear that having an elliptic solver which knows about mesh refinement is sort of hard. Is the multigrid solver of Eloisa Carpet aware? If so, I would probably elect to use it so long as it is designed to work for asymtotically flat initial data.
Yes, it is. If you want to set up your own solver grid, then you can easily use PETSc for this. If you want to use Carpet's AMR grids, then I would use Eloisa's solver. Let's make sure she also have time when you call in.
If not, I wonder whether it would be reasonable to take the output of a
solver done on unigrid and read it and remap it onto an AMR grid? Then run it as usual in ET. Either make the solver itself internally a Carpet aware thing or try to do it after the solver is done. Not an ideal way perhaps but perhaps a workaround.
That always works. However, if you add a few levels, then you lose much, both in accuracy (resolution), and due to the noise that interpolation adds.
-erik
On Thu, May 28, 2015 at 11:30 PM, Erik Schnetter schnetter@cct.lsu.edu
wrote:
On Wed, May 27, 2015 at 2:50 PM, Comer Duncan comer.duncan@gmail.com wrote:
Hi Erik,
Frank suggested you might be an appropriate person to ask about elliptic solver thorns that work in the current release of ET. I have, as you know, built Hilbert and have built the development version.
I am wanting to write a thorn which prepares a geon, approximated by initial data which has toroidal symmetry and is bounded. The desired distribution would be placed on the r axis in cylindrical coordinates and have a gaussian distribution. In the actual application, of course the initial data would be interpolated onto a Cartesian coordinate mesh or meshes. I just want to solve the Hamiltonian constraint and have the initial data be time-symmetric. This sounds like a pretty simple thing to do. I think I could start with Brill and put q to my purposes and pretty much get out what I want. However, the IDBrill does not seem to work. Also, I am wondering what kinds of elliptic solvers you know of that are fully supported in ET? Frank mentioned a petsc thorn. It does not get downloaded and built by default and looking at the einsteintoolkit.th file it is mentioned in:
# CactusElliptic thorns !TARGET = $ARR !TYPE = git !URL = https://bitbucket.org/cactuscode/cactuselliptic.git !REPO_PATH= $2 !CHECKOUT = CactusElliptic/EllPETSc CactusElliptic/TATPETSc CactusElliptic/EllBase #DISABLED CactusElliptic/EllPETSc CactusElliptic/EllSOR CactusElliptic/TATelliptic #DISABLED CactusElliptic/TATPETSc
However, what's with the DISABLED reference to EllPETSc is a bit concerning. So before I go though the exercise to look more into the EllPETSc, I thought I'd ask whether the EllPETSc thorn is fully compatible with Hilbert?
Can you thus please bring me up to the current moment on just what elliptic solvers are in use for initial data construction in Hilbert? I know many people have been focussing on the multiple BH problem, so there's not much traffic anymore about Brill. I do want to run Brill though and also do the "geon" problem.
Thanks for your advice and any pointers you can give.
Comer
The #DISABLED just means that it has been disabled, not that the thorn is bad. We only disable it because PETSc is a large package and may be difficult to install, and only very few people use PETSc with Cactus. I am regularly building with PETSc, and have also run the ET tests with PETSc enable on some of our machines.
I don't recommend using SOR. If you use mesh refinement, or if you use a machine more powerful than a laptop, then you can afford domains that are so large that SOR converges too slowly to be useful.
BAM_Elliptic is not freely available. It also has not been used in more than a decade, and I doubt it would be useful to resurrect it.
PETSc is a modern, full-featured elliptic solver. Unfortunately, there is no integration with Cactus's grid functions if you want to use mesh refinement. If you can live with a unigrid simulation, then EllPETSc allows you to solve linear elliptic equations. The thorns EllBase or EllPETSc should have some documentation on this; if not, maybe we can discuss next Monday during the ET telecon.
There is also a thorn TATPETSc that can solve non-linear elliptic equations (also unigrid only).
I haven't used these thorns in production in quite some time. Our initial conditions typically come from third-party solvers, e.g. Lorene or TwoPunctures.
The most modern solution is probably the multigrid solver that Eloisa wrote and that Ian described.
The situation of elliptic solvers in Cactus is not as good as I would like it to be. To help you out, it is probably best to talk in person, so that we see what you need. Do you have time next Monday to call in to our ET telecon?
-erik
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
On 29/05/15 17:12, Erik Schnetter wrote:
Yes, it is. If you want to set up your own solver grid, then you can easily use PETSc for this. If you want to use Carpet's AMR grids, then I would use Eloisa's solver. Let's make sure she also have time when you call in.
I confirm this. For necessity (and opportunism), my solver is very much Carpet-based.
Comer, did you have a go at any of the examples that come with the solver? The easier way to get started is to run one of the cases under CT_MultiLevel/par/, and then try and tweak it to meet your needs.
I will be available Monday, June 8th after about 11:00 CEST.
Eloisa
users@lists.einsteintoolkit.org
-
Comer Duncan -
Eloisa Bentivegna -
Erik Schnetter -
Ian Hinder