Re: [Users] Question about IllinoisGRMHD C2P failure and kuibit Psi4 extraction

Dear Taiki,

This is an experimental response prepared using a set of LLM agents and as a test of the newly created GRHayLET knowledge base: https://github.com/GRHayL/GRHayLET/blob/main/AGENTS.md

Zach will be traveling for the next month and hopes that this response will be helpful in the meantime.

The single Psi4 value is consistent with the simulation stopping at iteration 0. In kuibit, `SimDir.gws[100]` selects the extraction radius r=100, and `[(2,2)]` selects one spherical-harmonic mode. The resulting `.x` and `.y` arrays are the time and complex data samples for that particular radius and mode; they are not containers for all radii or modes. You can inspect what was read with

print(SimDir.gws.radii) print(SimDir.gws[100].available_lm) print(SimDir.gws[100][(2,2)].x)

Since the last command returns only t=0 and the Psi4 ASCII files contain only one row, kuibit appears to be reading the available output correctly. With `Multipole::out_every = 128`, the next sample would be written at iteration 128; the attached parameter file instead uses 64, so its next sample would be at iteration 64. Neither setting can produce another sample if the evolution never advances past iteration 0. Files containing spatial profiles can have many rows while still representing only iteration 0.

The IllinoisGRMHD message does report C2P recovery failures, but it is an INFO message and does not by itself explain why Cactus stopped:

* `Averaged pts = 38250 Font1D 38250` says that 38,250 points first required neighbor-averaging attempts and then reached the Font1D fallback. * `Failures: 2250` says that recovery still failed at 2,250 points and those points were reset to atmosphere. * These are counters for the local grid data handled by that printed C2P invocation, not necessarily global counts over the whole simulation. * `tau nan` and the NaN momentum diagnostics mean that an aggregate post-recovery error calculation encountered non-finite values. They do not imply that every value of tau or every momentum component is NaN.

Thus, there are genuine C2P failures, but C2P may be reacting to invalid metric, primitive, EOS, or conservative input rather than creating the first NaN. The first priority is to distinguish the reason for termination from the first source of non-finite data.

First, please inspect the batch-system exit status and the final 100--200 lines of both stdout and stderr. Search backward for the termination request, especially `WARNING level 0`, `CCTK_ERROR`, NaNChecker, TerminationTrigger, MPI abort, or scheduler kill messages; an unrelated earlier warning need not explain the stop. The attached parameter file also enables `TerminationTrigger::termination_from_file` with the file name `TERMINATE`. Please check for a stale `TERMINATE` file in the run/output directory. Such a file can request a clean stop at iteration 0 even if the C2P INFO message did not terminate the run.

There also appears to be a mismatch between the quoted diagnostic and the requested recovery configuration. If the quoted text is the complete diagnostic block, a line containing `Font1D` but neither entropy nor Y_e resembles the Hybrid non-entropy path in current IllinoisGRMHD. In contrast, the attached file requests a tabulated EOS with entropy evolution and three backup routines. The TabulatedEntropy diagnostic includes entropy and Y_e and does not print that `Font1D` field:

https://github.com/GRHayL/GRHayLET/blob/301648c62bb87544adc7399d3239fef97ae0...

https://github.com/GRHayL/GRHayLET/blob/301648c62bb87544adc7399d3239fef97ae0...

Please check the complete diagnostic block, the startup schedule, and the parameter file copied into the simulation output directory. In particular:

1. Run once in a fresh output directory with `IOUtil::recover = "no"`, or otherwise disable recovery explicitly, and ensure that no stale termination file is present. A fresh output directory alone is insufficient here because the attached `IOUtil::recover_dir` names a separate `output_directory_01/checkpoints` directory. 2. Confirm that the runtime parameter file contains

GRHayLib::EOS_type = "Tabulated" GRHayLib::evolve_entropy = yes GRHayLib::con2prim_routine = "Palenzuela1D"

3. Check whether `ID_converter_ILGRMHD` is active and whether startup prints `Running GRHayLib with backward-compatible settings`. In current IllinoisGRMHD, activating that thorn adds legacy Hybrid scheduling and initializes Noble2D/Font1D compatibility settings independently of the modern EOS-selected schedule, potentially creating conflicting paths. For a modern GRHayLib/GRHayLID setup, remove it and confirm that only the TabulatedEntropy C2P routine is scheduled. The attached file does not list it, making this check especially useful for detecting submission of a different generated parameter file or executable. 4. Keep `Carpet::ghost_size = 3`, as already specified in the attached file. IllinoisGRMHD requires at least three ghost zones, and increasing this to four will not repair invalid C2P input. 5. Leave the Multipole interpolation settings aside until the evolution advances. Multipole analysis cannot repair an iteration-0 Prim2Con/C2P problem.

For a more direct NaN check, add `NaNChecker` to `ActiveThorns` and use something like

NaNChecker::check_after = 0 NaNChecker::check_every = 1 NaNChecker::check_for = "both" NaNChecker::action_if_found = "just warn" NaNChecker::verbose = "all" NaNChecker::check_vars = " ADMBase::metric ADMBase::lapse ADMBase::shift HydroBase::rho HydroBase::press HydroBase::eps HydroBase::entropy HydroBase::Y_e HydroBase::temperature IllinoisGRMHD::grmhd_velocities IllinoisGRMHD::u0 IllinoisGRMHD::Ax IllinoisGRMHD::Ay IllinoisGRMHD::Az IllinoisGRMHD::phitilde IllinoisGRMHD::grmhd_B_stagger IllinoisGRMHD::grmhd_B_center IllinoisGRMHD::grmhd_conservatives IllinoisGRMHD::ent_star IllinoisGRMHD::Ye_star "

NaNChecker reports fields at its scheduled check point; that timing depends on the NaNChecker version and does not prove which routine first produced a non-finite value. To distinguish invalid initial primitives from Prim2Con output and C2P effects, check immediately before Prim2Con, immediately after Prim2Con but before C2P, and immediately after C2P. The NaNChecker documentation is:

https://einsteintoolkit.org/thornguide/CactusUtils/NaNChecker/documentation....

A finite rho diagnostic together with non-finite tau and momentum only weakly narrows the cause. GRHayL computes the conserved density from fewer inputs, whereas conserved energy and momentum use additional thermodynamic, velocity, metric, and magnetic quantities:

https://github.com/GRHayL/GRHayL/blob/8ef7e60884360ed4e6bef52c1be8fcff4558f3...

I would therefore check lapse and metric components, velocity/Lorentz factor, pressure, specific internal energy, temperature, Y_e, and vector-potential or magnetic-field components at iteration 0. Although the attached file sets `Seed_Magnetic_Fields::A_b = 0`, that setting alone does not prove that every magnetic array is initialized and finite, so I would not rank the EOS and magnetic possibilities until those checks are available.

Please also report the Einstein Toolkit release and the exact GRHayL and GRHayLET commit IDs used to build the executable. Older combinations may predate relevant fixes. For example, GRHayLET pull request 14 fixed missing error-control flow in the HybridEntropy, Tabulated, and TabulatedEntropy recovery branches (not the Hybrid branch):

https://github.com/GRHayL/GRHayLET/pull/14

Finally, the current Gallery `bns.par` uses GRHydro with an ideal-fluid EOS, not IllinoisGRMHD with an SFHo table. Changing the hydrodynamics thorn and EOS is therefore a substantial new configuration rather than a small change to a validated Gallery setup:

https://einsteintoolkit.org/gallery/bns/index.html

https://einsteintoolkit.org/gallery/bns/bns.par

The most useful items to send next would be the complete iteration-0 C2P diagnostic block, the first warning/error and final output lines, the batch exit status, the copied runtime parameter file, and the build commit IDs.

Best regards,

-Zach

* * * Zachariah Etienne Prof. of Physics, U. of Idaho Adjunct Prof. of Physics & Astronomy, West Virginia U. https://etienneresearch.com https://blackholesathome.net

On Fri, Jul 3, 2026 at 3:10 AM MAKINO Taiki via Users < users@einsteintoolkit.org> wrote:

Dear Jordan

Thank you for your quick response.

I understand what I should do. I will wait for feedback from the others, and I consider attending the weekly meeting.

Thank you again for your advice.

Best regards,

Taiki Makino Graduate School of Science, Nagoya University *makino*.taiki.g6@s.mail.nagoya-u.ac.jp makino.taiki.g6@s.mail.nagoya-u.ac.jp

取得 Outlook for Mac https://aka.ms/GetOutlookForMac *差出人: *Jordan Nicoules jnicoules@ua.pt *日付: *金曜日, 2026年7月3日 18:38 *宛先: *MAKINO Taiki makino.taiki.g6@s.mail.nagoya-u.ac.jp *CC: *users@einsteintoolkit.org users@einsteintoolkit.org *件名: *Re: Question about IllinoisGRMHD C2P failure and kuibit Psi4 extraction

Hi Taiki,

(apologies for my previous typo)

The next step would be to understand why the simulation crashes and doesn't go past t=0. This is beyond what I can quickly hotfix I think, so *I'm calling to other members to answer that.* Potentially, if you have the chance, the weekly calls can also be an opportunity to interact with people directly and put forth your questions.

Regarding files that have more than one line of data, that would make sense for the 1D profiles, but they still probably only correspond to iteration 0 (t=0).

Best,

Jordan

---

*From:* MAKINO Taiki makino.taiki.g6@s.mail.nagoya-u.ac.jp *Sent:* Friday, July 3, 2026 10:14:57 AM *To:* Jordan Nicoules *Cc:* users@einsteintoolkit.org *Subject:* Re: Question about IllinoisGRMHD C2P failure and kuibit Psi4 extraction

*CUIDADO:* Email de um sistema externo. Cuidado com links, anexos e pedidos de dados/senhas. *CAUTION:* Email from an external system. Be careful with links, attachments, and requests for data/passwords. Dear Jordan

Thank you for your response.

When I checked the result of my simulation, SimDir.gws[100][(2,2)].x was indeed 0 as you said and the simulation stopped at iteration=0.

Also, I checked the psi4 ASCII files and all of them contained only one line. (However, the ASCII files that do not contain information of psi4 have several lines of data.)

I am not sure what is causing this. What should I do next ?

Best regards,

Taiki Makino

Graduate School of Science, Nagoya University

*makino*.taiki.g6@s.mail.nagoya-u.ac.jp makino.taiki.g6@s.mail.nagoya-u.ac.jp

取得 Outlook for Mac https://aka.ms/GetOutlookForMac *差出人: *Jordan Nicoules jnicoules@ua.pt *日付: *木曜日, 2026年7月2日 19:31 *宛先: *MAKINO Taiki makino.taiki.g6@s.mail.nagoya-u.ac.jp; users@einsteintoolkit.org users@einsteintoolkit.org *件名: *Re: Question about IllinoisGRMHD C2P failure and kuibit Psi4 extraction

Hi Taiko,

I will (partly) skip the first part of your message and let others reply on that, as I don't have experience with NS simulations.

Regarding the gravitational wave extraction, I think you are accessing the data correctly (your example gives the time series for one mode at one radius), and there should be more than one element in general. From your parameter file, it seems the output should go to an ascii file (it's the default), so you can double check in that file too.

However, since you have `Multipole::out_every = 128`, does your simulation reach iteration 128 and later? In particular, because you're getting nans at iteration 0, does the simulation actually go past t=0? Given what you're saying, I guess your single element of `SimDir.gws[100][(2,2)].x` is 0 (i.e t=0).

Best,

Jordan

---

*From:* Users users-bounces@einsteintoolkit.org on behalf of MAKINO Taiki via Users users@einsteintoolkit.org *Sent:* Thursday, July 2, 2026 12:39:00 AM *To:* users@einsteintoolkit.org *Subject:* [Users] Question about IllinoisGRMHD C2P failure and kuibit Psi4 extraction

*CUIDADO:* Email de um sistema externo. Cuidado com links, anexos e pedidos de dados/senhas. *CAUTION:* Email from an external system. Be careful with links, attachments, and requests for data/passwords. Dear Einstein Toolkit Users,

I am a new Einstein Toolkit user, and I have some questions about a BNS simulation.

I am trying to run a BNS simulation using bns.par, which I downloaded from Einstein Toolkit Gallery ( https://einsteintoolkit.org/gallery/bns/index.html) and slightly modified.

I can successfully create and submit the simulation but the following message appears during the simulation:

INFO (IllinoisGRMHD): C2P: Iter. # 0, Lev: 5 NumPts= 132300 | Backups: 0 0 0 | Fixes: VL= 0 rho*= 0 Averaged pts = 38250 Font1D 38250 | Failures: 2250 InHoriz= 0 / 0 | 1.00 iters/gridpt Error, Sum: rho 6.378e-08, 2.433e-05 | tau nan, -nan | Sx nan, nan | Sy nan, nan | Sz nan, nan

To investigate this problem, I added the following parameters: Multipole::interpolator_name = "Lagrange polynomial interpolation" Multipole::interpolator_pars = "order=4"

I also changed the parameter Driver::ghost_size from 3 to 4, but the same error occurred.

I would like to ask:

• What could cause tau, Sx, Sy, and Sz to become NaN in this situation?
• Does this indicate a failure of C2P conversion?
• Are there any parameter settings or initial data that I should check

first?

Additionally, I have another question. When I use kuibit to extract Psi4, I expected to obtain the full time series (or all available modes) but the returned object contains only one element, for example, SimDir.gws[100][(2,2)].y=[-9.79586237e-15-5.27476073e-14j].

Other attributes such as SimDir.gws[100][(2,2)].x

also contain only one element. Is this a correct behavior or could it indicate a problem with my .par file?

For reference, I have attached my parameter file.

Thank you very much for your time and any suggestions.

Best regards,

Taiki Makino

Graduate School of Science, Nagoya University

*makino*.taiki.g6@s.mail.nagoya-u.ac.jp makino.taiki.g6@s.mail.nagoya-u.ac.jp

取得 Outlook for Mac https://aka.ms/GetOutlookForMac

取得 Outlook for Mac https://aka.ms/GetOutlookForMac _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users