Hi Erik,

Great questions.

- Is it necessary to have an explicit conversion thorn? Couldn't these routines be simply part of IllinoisGRMHD? For example, McLachlan uses its own variable, and contains its own routines to convert from/to ADMBase.

IllinoisGRMHD is designed to maximize user-friendliness, and to this end one of my goals is to minimize the amount of source code inside IllinoisGRMHD that is not directly related to GRMHD evolution. Thus if possible, I would prefer not incorporating functionality from these thorns into IllinoisGRMHD.

- If the routines really need to exist in two thorns, then I suggest having the actual routines in only one place, and calling the routines from the other place. (I probably completely missed the original motivation.) If necessary, you would need to introduce a third thorn containing these routines that these two thorns can call.

I would prefer not adding another thorn if possible, as I want to minimize the effort required by a new user to read and understand the IllinoisGRMHD evolution code. If they need to hunt in other thorns for needed functions, learning IllinoisGRMHD would require more effort. I could also raise a performance objection, in that the compiler may not be able to optimize as well if functions deep within loops exist in other thorns.

Here's a promising alternative: I just looked at the two compatibility-layer thorns in a bit more detail. Turns out there are a total of 2 functions from within IllinoisGRMHD that are needed by these two compatibility-layer thorns. If I make these two functions share-able in IllinoisGRMHD/interface.ccl, they can be accessed from the other thorns, right? Two complications: 1) these functions depend on a number of other functions within IllinoisGRMHD, so would this linkage work? 2) there is a header file "IllinoisGRMHD_headers.h" that is needed by both thorns...

In the end, as described in the IllinoisGRMHD trac ticket ( https://trac.einsteintoolkit.org/ticket/1796) I would like to move IllinoisGRMHD to a wider arrangement (called WVUThorns, for example), because it will be the first of many thorns my research team and I intend to contribute to the Toolkit. So if all these IllinoisGRMHD-related thorns exist in the same arrangement, maybe symbolic links aren't the worst possible option. git seems to have no problem handling them.

- Would scheduling "IN initial AFTER HydroBase_Initial" work?

Not sure; I will need to try this.

- Similarly, there should be a group HydroBase_SetHydroBaseVars where the respective routine should be scheduled. I think half of the functionality that is present in ADMBase is still missing from HydroBase since, so far, there was only one thorn (... or two very similar thorns ...) that used HydroBase. Feel free to suggest the respective extensions to HydroBase, modelled after ADMBase if possible.

This is a neat idea, and brings up another item on my TODO list: HydroBase currently does not allow for any input as to the staggering of the A-fields. Currently IllinoisGRMHD must define its own A-field variables in individual variable groups because the prolongation type for Avec in HydroBase is not independently specifiable for the different components of A. In addition to adding a group HydroBase_SetHydroBaseVars, it would also be nice to generalize the specification of A-fields.

Thanks, and have a great afternoon.

-Zach

* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University

On Mon, Aug 17, 2015 at 12:47 PM, Erik Schnetter schnetter@cct.lsu.edu wrote:

Zach

Apologies for missing half of the discussion this morning -- I had an unexpected visitor in my office.

• Is it necessary to have an explicit conversion thorn? Couldn't these

routines be simply part of IllinoisGRMHD? For example, McLachlan uses its own variable, and contains its own routines to convert from/to ADMBase.

• If the routines really need to exist in two thorns, then I suggest

having the actual routines in only one place, and calling the routines from the other place. (I probably completely missed the original motivation.) If necessary, you would need to introduce a third thorn containing these routines that these two thorns can call.

• Symbolic links will in general not work since they will lead to two

different routines with the same name, something that confuses the Unix linker and may lead to random problems (segfaults, undefined symbols at link time, etc.) If identical code exists in different thorns, then the routines must have different names.

• Would scheduling "IN initial AFTER HydroBase_Initial" work?

• Similarly, there should be a group HydroBase_SetHydroBaseVars where the

respective routine should be scheduled. I think half of the functionality that is present in ADMBase is still missing from HydroBase since, so far, there was only one thorn (... or two very similar thorns ...) that used HydroBase. Feel free to suggest the respective extensions to HydroBase, modelled after ADMBase if possible.

-erik

On Mon, Aug 17, 2015 at 11:49 AM, Zach Etienne zachetie@gmail.com wrote:

...

Hi all,

In conjunction with the recent update to IllinoisGRMHD, I just made public a couple of thorns that provide a HydroBase <-> IllinoisGRMHD compatibility layer.

ID_converter_ILGRMHD: This thorn converts Valencia formulation variables into variables IllinoisGRMHD can read, making it possible to input HydroBase-compatible initial data into IllinoisGRMHD.

convert_to_HydroBase: This thorn converts IllinoisGRMHD GRMHD variables into variables compatible with HydroBase every "convert_to_HydroBase_every" iterations. This thorn enables you to use HydroBase-compatible analysis thorns with IllinoisGRMHD.

There are a couple items on the TODO list for these thorns:

1. Because there is no uniform scheduling bin that runs after

HydroBase_Initial, one must edit ID_converter_ILGRMHD/schedule.ccl to force the ID converter to be run AFTER the final initial data function in HydroBase_Initial

2. As we discussed during the Aug 17, 2015 ET telecon, there are symlinks

inside these thorns to IllinoisGRMHD source code, requiring these thorns to be placed in the same subdirectory as the main IllinoisGRMHD code. If the symlinks must go, I would like to know the least painful way of getting rid of these symlinks without duplicating code.

Here's a link to the bitbucket repo: https://bitbucket.org/zach_etienne/illinoisgrmhd_compatibilitylayer

-Zach

---

Zachariah Etienne Assistant Professor of Mathematics West Virginia University

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/

Hi all,

Using Carpet/CarpetLib as an example, I was able to figure out how to remove the nasty symlinks between IllinoisGRMHD and the IllinoisGRMHD_CompatibilityLayer thorns (see latest commits).

Special thanks to Erik for pointing me in the right direction!

Now to address Erik's questions:

On Mon, Aug 17, 2015 at 11:10 PM, Erik Schnetter schnetter@cct.lsu.edu wrote:

On Mon, Aug 17, 2015 at 2:23 PM, Zach Etienne zachetie@gmail.com wrote:

...

Hi Erik,

Great questions.

• Is it necessary to have an explicit conversion thorn? Couldn't these

routines be simply part of IllinoisGRMHD? For example, McLachlan uses its own variable, and contains its own routines to convert from/to ADMBase.

IllinoisGRMHD is designed to maximize user-friendliness, and to this end one of my goals is to minimize the amount of source code inside IllinoisGRMHD that is not directly related to GRMHD evolution. Thus if possible, I would prefer not incorporating functionality from these thorns into IllinoisGRMHD.

That's fine. However, what I then don't understand -- why is the same function apparently needed both inside and outside of IllinoisGRMHD?

There are two functions that are needed inside & outside IllinoisGRMHD: Function A) A function that computes the BSSN variables from the ADMBase variables. Function B) A function that checks the physicality of the IllinoisGRMHD primitive variables and then computes the IllinoisGRMHD conservative variables from the IllinoisGRMHD primitive variables (prim2con).

To give you a better understanding of why these functions are needed both inside & outside IllinoisGRMHD, here is the order of operations OUTSIDE IllinoisGRMHD within the initial data converter function (in the compatibility layer thorn ID_converter_ILGRMHD). Note that when this thorn is called, only HydroBase and ADMBase variables have been set. 1) [Function A] 2) Convert the HydroBase primitive variables (Valencia formulation) into IllinoisGRMHD primitive variables. 3) Compute B from A (depends on Function A) 4) [Function B] (depends on Function A)

[Function A] is needed in IllinoisGRMHD in multiple places, to use the latest, updated ADMBase variables during an evolution. [Function B] is needed in IllinoisGRMHD's con2prim, so that if the primitive or conservative variables fall outside the known physical range, we can then compute prim2con so the primitives & conservatives are consistent.

• If the routines really need to exist in two thorns, then I suggest

...

having the actual routines in only one place, and calling the routines from the other place. (I probably completely missed the original motivation.) If necessary, you would need to introduce a third thorn containing these routines that these two thorns can call.

I would prefer not adding another thorn if possible, as I want to minimize the effort required by a new user to read and understand the IllinoisGRMHD evolution code. If they need to hunt in other thorns for needed functions, learning IllinoisGRMHD would require more effort. I could also raise a performance objection, in that the compiler may not be able to optimize as well if functions deep within loops exist in other thorns.

Here's a promising alternative: I just looked at the two compatibility-layer thorns in a bit more detail. Turns out there are a total of 2 functions from within IllinoisGRMHD that are needed by these two compatibility-layer thorns. If I make these two functions share-able in IllinoisGRMHD/interface.ccl, they can be accessed from the other thorns, right? Two complications:

1. these functions depend on a number of other functions within

IllinoisGRMHD, so would this linkage work? 2) there is a header file "IllinoisGRMHD_headers.h" that is needed by both thorns...

Cactus has a mechanism for this. This will just be a regular function call, so the fact that these two functions call other functions doesn't matter. If you are using a header file, then you don't need to additionally declare the function in interface.ccl. The interface.ccl mechanism is only needed if you want flexibility, so that one can change the thorns that provide these functions at run time. That doesn't seem relevant here.

Thanks to your suggestion, I figured this out. No symlinks necessary anymore. Thanks!

...

In the end, as described in the IllinoisGRMHD trac ticket ( https://trac.einsteintoolkit.org/ticket/1796) I would like to move IllinoisGRMHD to a wider arrangement (called WVUThorns, for example), because it will be the first of many thorns my research team and I intend to contribute to the Toolkit. So if all these IllinoisGRMHD-related thorns exist in the same arrangement, maybe symbolic links aren't the worst possible option. git seems to have no problem handling them.

As I mentioned, symbolic links give you the same file, and hence the same function name multiple times. This is not allowed by the linker. It is also confusing for people to see the same file / functions in two different places. Please try to find another solution. Regular function calls to another thorn will just work (if you provide respective header files), and there is no overhead associated with this.

• Would scheduling "IN initial AFTER HydroBase_Initial" work?

...

Not sure; I will need to try this.

I have still not yet had a chance to try this. As a stopgap, I have looked over a few initial data schedule.ccl files and modified ID_converter_ILGRMHD's schedule.ccl file to be appropriately scheduled after those thorns have completed setting up initial data.

...

• Similarly, there should be a group HydroBase_SetHydroBaseVars where the

respective routine should be scheduled. I think half of the functionality that is present in ADMBase is still missing from HydroBase since, so far, there was only one thorn (... or two very similar thorns ...) that used HydroBase. Feel free to suggest the respective extensions to HydroBase, modelled after ADMBase if possible.

This is a neat idea, and brings up another item on my TODO list: HydroBase currently does not allow for any input as to the staggering of the A-fields. Currently IllinoisGRMHD must define its own A-field variables in individual variable groups because the prolongation type for Avec in HydroBase is not independently specifiable for the different components of A. In addition to adding a group HydroBase_SetHydroBaseVars, it would also be nice to generalize the specification of A-fields.

Can you make a concrete suggestion?

Given the magnitude of my proposed commit into the ET at the moment (IllinoisGRMHD and compatibility layer thorns) and my very limited time now that the semester has started, I would like to postpone this discussion until after the next ET release. Currently I have a private thorn "seed_magnetic_fields", which ignores HydroBase variables altogether when setting (staggered) A fields. Unless a better thorn exists for setting up generic magnetic field configurations from the vector potential, it would be nice to extend this thorn to use unstaggered A fields as well. I will release this thorn to the wild once IllinoisGRMHD & related thorns are accepted into the ET.

Hi Ian,

Just a quick comment: maybe it would be possible to isolate all such

"extra" functionality in a single source file, and make it clear that this is only present for compatibility, and isn't relevant to the internal workings of the thorn. It is true that

compatibility code can make things cluttered, but maybe having it

separate would be enough?

Thanks for your comment.

Using Carpet/CarpetLib thorns as a template, I was able to figure out how to remove the symlinks. In my previous email, I define [Function A] and [Function B]. Regarding splitting this functionality into a single source file,

* [Function A] exists within a source file called, appropriately enough, "convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C" * [Function B] exists next to a function that applies a fix ("apply_tau_floor") to the conservative variables, and both exist within a source file called: "apply_tau_floor__enforce_limits_on_primitives_and_recompute_conservs.C"

Now that the symlinks issue has been resolved, I would be happy to hear any suggestions or comments you have regarding the current IllinoisGRMHD & related thorns.

-Zach

* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University

On Tue, Aug 18, 2015 at 5:48 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:

On 17 Aug 2015, at 20:23, Zach Etienne zachetie@gmail.com wrote:

Hi Erik,

Great questions.

• Is it necessary to have an explicit conversion thorn? Couldn't these

routines be simply part of IllinoisGRMHD? For example, McLachlan uses its own variable, and contains its own routines to convert from/to ADMBase.

IllinoisGRMHD is designed to maximize user-friendliness, and to this end one of my goals is to minimize the amount of source code inside IllinoisGRMHD that is not directly related to GRMHD evolution. Thus if possible, I would prefer not incorporating functionality from these thorns into IllinoisGRMHD.

Just a quick comment: maybe it would be possible to isolate all such "extra" functionality in a single source file, and make it clear that this is only present for compatibility, and isn't relevant to the internal workings of the thorn. It is true that compatibility code can make things cluttered, but maybe having it separate would be enough?

-- Ian Hinder http://members.aei.mpg.de/ianhin