Dear ET folks,

After months of work, I am pleased to release my rewrite of the Illinois numerical relativity group's staggered A-field GRMHD thorn to the ET community. The new thorn is called IllinoisGRMHD, and yields results identical (within roundoff error) to the original GRMHD code of the Illinois group with AMR & unigrids, with bitant symmetry & without, and with single or multiple MPI processes. We used precisely these GRMHD algorithms to produce our latest GRMHD-based papers.

IllinoisGRMHD is written entirely in C, except for some C++ file I/O for debugging purposes, and all loops are OpenMP-compatible. The coding standard is C99.

I would like to thank Frank and the CCT folks, who have graciously agreed to host IllinoisGRMHD on the ET SVN server, as the thorn awaits maintainers' approval for incorporation into the official ET.

Download IllinoisGRMHD via the following command: svn co https://svn.einsteintoolkit.org/cactus/EinsteinEvolve/IllinoisGRMHD/trunk/

Browse IllinoisGRMHD source code in your web browser via: https://svn.einsteintoolkit.org/cactus/EinsteinEvolve/IllinoisGRMHD/trunk/

** Background ** Roughly a decade of work has gone into the original code, and nearly all of the credit for debugging that battle-hardened code goes to Stu Shapiro and other former & current members of his group, to whom I give my great thanks and send the following message: Despite weeks entirely devoted to debugging this rewritten code, I found no major bugs in the original code.

** License ** The IllinoisGRMHD thorn is hereby released under the GNU GPL v2 or higher, to maximize compatibility with the ET thorns.

** Changes from Original Code ** The IllinoisGRMHD thorn contains significantly fewer lines than the code on which it is based, for two reasons:

1) Functionality that is no longer used (mostly dead code at this point) has been removed, and

2) New data structures in the code make it more compact, reducing the total number of lines in the reconstruction & MHD RHS evaluation routines to ~1200. I hope that the data structures in the new code will make it easier to read, less intimidating for beginners, and much easier to modify & extend, to do new science.

I started development of IllinoisGRMHD as a standalone code, to minimize the turnaround time in the first round of debugging and to try new ideas for enhancing the performance, as well as making the code more compact and easier to read. The standalone driver routines still exist, but bugfixes found in the "thorn version" have not yet been backported to these routines. I am happy to report that the new code is significantly faster than the original code.

** Conservative-to-Primitives Routines ** We previously released our Conservative-to-Primitives routines to the ET mailing list, based on the very nice 2D HARM primitive solver of Scott Noble et al. An updated version is included in this release.

** Future Work ** Now that the IllinoisGRMHD thorn has been released, there is still some work to do, summarized below.

1) If using AMR grids, this thorn requires support for staggered prolongation & restriction algorithms within Carpet. We open-sourced these algorithms for an old version of Carpet back in March of 2012. Since then, Carpet has moved forward and these original algorithms (which we still use in our old version of Carpet) are no longer compatible with the latest ET. Philipp Moesta informed me over the summer that he has ported these operators to a more recent Carpet, and I will be working with him in the coming weeks to incorporate these into the latest Carpet.

2) This thorn currently exists as a drop-in replacement for the original GRMHD thorn of the Illinois group. I will be working in the coming weeks to make it more like a drop-in replacement for GRHydro instead, though perhaps someday it could be incorporated into GRHydro.

3) There are several minor to-do tasks remaining in the code, marked within comments labeled "FIXME" (grep for it).

4) There exist several possible avenues for code optimization, such as some possibly extraneous SYNC's within the schedule.ccl file.

5) More and better comments that clarify what the code is doing will be incorporated.

6) I am unsure how strongly this code violates coding standards & practices. E.g., I sometimes break lines according to the width of my laptop screen. Also, I love variable & function names that explain *exactly* what they do, which for some (possibly good) reason is frowned upon by other C/C++ programmers. I personally find that long names make it easier to read & understand a code years later.

7) I apologize for the lack of full written documentation at this point, but promise that it will be forthcoming. In the meantime, I encourage you to look at the large number of comments in the code, and if you find a point of confusion, please email me, and I'll fix it!

-Zach