[Users] Question about IllinoisGRMHD C2P failure and kuibit Psi4 extraction
Zach Etienne
zachetie at gmail.com
Thu Jul 23 10:33:03 CDT 2026
Dear Taiki,
This is an experimental response prepared using a set of LLM agents and as a
test of the newly created GRHayLET knowledge base:
https://github.com/GRHayL/GRHayLET/blob/main/AGENTS.md
Zach will be traveling for the next month and hopes that this response will
be helpful in the meantime.
The single Psi4 value is consistent with the simulation stopping at
iteration
0. In kuibit, `SimDir.gws[100]` selects the extraction radius r=100, and
`[(2,2)]` selects one spherical-harmonic mode. The resulting `.x` and `.y`
arrays are the time and complex data samples for that particular radius and
mode; they are not containers for all radii or modes. You can inspect what
was
read with
print(SimDir.gws.radii)
print(SimDir.gws[100].available_lm)
print(SimDir.gws[100][(2,2)].x)
Since the last command returns only t=0 and the Psi4 ASCII files contain
only
one row, kuibit appears to be reading the available output correctly. With
`Multipole::out_every = 128`, the next sample would be written at iteration
128; the attached parameter file instead uses 64, so its next sample would
be
at iteration 64. Neither setting can produce another sample if the evolution
never advances past iteration 0. Files containing spatial profiles can have
many rows while still representing only iteration 0.
The IllinoisGRMHD message does report C2P recovery failures, but it is an
INFO
message and does not by itself explain why Cactus stopped:
* `Averaged pts = 38250 Font1D 38250` says that 38,250 points first required
neighbor-averaging attempts and then reached the Font1D fallback.
* `Failures: 2250` says that recovery still failed at 2,250 points and those
points were reset to atmosphere.
* These are counters for the local grid data handled by that printed C2P
invocation, not necessarily global counts over the whole simulation.
* `tau nan` and the NaN momentum diagnostics mean that an aggregate
post-recovery error calculation encountered non-finite values. They do not
imply that every value of tau or every momentum component is NaN.
Thus, there are genuine C2P failures, but C2P may be reacting to invalid
metric, primitive, EOS, or conservative input rather than creating the first
NaN. The first priority is to distinguish the reason for termination from
the
first source of non-finite data.
First, please inspect the batch-system exit status and the final 100--200
lines
of both stdout and stderr. Search backward for the termination request,
especially `WARNING level 0`, `CCTK_ERROR`, NaNChecker, TerminationTrigger,
MPI abort, or scheduler kill messages; an unrelated earlier warning need not
explain the stop. The attached parameter file also enables
`TerminationTrigger::termination_from_file` with the file name `TERMINATE`.
Please check for a stale `TERMINATE` file in the run/output directory. Such
a
file can request a clean stop at iteration 0 even if the C2P INFO message
did
not terminate the run.
There also appears to be a mismatch between the quoted diagnostic and the
requested recovery configuration. If the quoted text is the complete
diagnostic block, a line containing `Font1D` but neither entropy nor Y_e
resembles the Hybrid non-entropy path in current IllinoisGRMHD. In contrast,
the attached file requests a tabulated EOS with entropy evolution and three
backup routines. The TabulatedEntropy diagnostic includes entropy and Y_e
and
does not print that `Font1D` field:
https://github.com/GRHayL/GRHayLET/blob/301648c62bb87544adc7399d3239fef97ae0e5d4/IllinoisGRMHD/src/Hybrid/conservs_to_prims.c#L334-L354
https://github.com/GRHayL/GRHayLET/blob/301648c62bb87544adc7399d3239fef97ae0e5d4/IllinoisGRMHD/src/TabulatedEntropy/conservs_to_prims.c#L337-L361
Please check the complete diagnostic block, the startup schedule, and the
parameter file copied into the simulation output directory. In particular:
1. Run once in a fresh output directory with `IOUtil::recover = "no"`, or
otherwise disable recovery explicitly, and ensure that no stale
termination
file is present. A fresh output directory alone is insufficient here
because the attached `IOUtil::recover_dir` names a separate
`output_directory_01/checkpoints` directory.
2. Confirm that the runtime parameter file contains
GRHayLib::EOS_type = "Tabulated"
GRHayLib::evolve_entropy = yes
GRHayLib::con2prim_routine = "Palenzuela1D"
3. Check whether `ID_converter_ILGRMHD` is active and whether startup prints
`Running GRHayLib with backward-compatible settings`. In current
IllinoisGRMHD, activating that thorn adds legacy Hybrid scheduling and
initializes Noble2D/Font1D compatibility settings independently of the
modern EOS-selected schedule, potentially creating conflicting paths. For
a modern GRHayLib/GRHayLID setup, remove it and confirm that only the
TabulatedEntropy C2P routine is scheduled. The attached file does not
list
it, making this check especially useful for detecting submission of a
different generated parameter file or executable.
4. Keep `Carpet::ghost_size = 3`, as already specified in the attached file.
IllinoisGRMHD requires at least three ghost zones, and increasing this to
four will not repair invalid C2P input.
5. Leave the Multipole interpolation settings aside until the evolution
advances. Multipole analysis cannot repair an iteration-0 Prim2Con/C2P
problem.
For a more direct NaN check, add `NaNChecker` to `ActiveThorns` and use
something like
NaNChecker::check_after = 0
NaNChecker::check_every = 1
NaNChecker::check_for = "both"
NaNChecker::action_if_found = "just warn"
NaNChecker::verbose = "all"
NaNChecker::check_vars = "
ADMBase::metric ADMBase::lapse ADMBase::shift
HydroBase::rho HydroBase::press HydroBase::eps
HydroBase::entropy HydroBase::Y_e HydroBase::temperature
IllinoisGRMHD::grmhd_velocities
IllinoisGRMHD::u0
IllinoisGRMHD::Ax IllinoisGRMHD::Ay IllinoisGRMHD::Az
IllinoisGRMHD::phitilde
IllinoisGRMHD::grmhd_B_stagger
IllinoisGRMHD::grmhd_B_center
IllinoisGRMHD::grmhd_conservatives
IllinoisGRMHD::ent_star IllinoisGRMHD::Ye_star
"
NaNChecker reports fields at its scheduled check point; that timing depends
on
the NaNChecker version and does not prove which routine first produced a
non-finite value. To distinguish invalid initial primitives from Prim2Con
output and C2P effects, check immediately before Prim2Con, immediately after
Prim2Con but before C2P, and immediately after C2P. The NaNChecker
documentation is:
https://einsteintoolkit.org/thornguide/CactusUtils/NaNChecker/documentation.html
A finite rho diagnostic together with non-finite tau and momentum only
weakly
narrows the cause. GRHayL computes the conserved density from fewer inputs,
whereas conserved energy and momentum use additional thermodynamic,
velocity,
metric, and magnetic quantities:
https://github.com/GRHayL/GRHayL/blob/8ef7e60884360ed4e6bef52c1be8fcff4558f3f5/GRHayL/Con2Prim/compute_conservs.c#L92-L98
I would therefore check lapse and metric components, velocity/Lorentz
factor,
pressure, specific internal energy, temperature, Y_e, and vector-potential
or
magnetic-field components at iteration 0. Although the attached file sets
`Seed_Magnetic_Fields::A_b = 0`, that setting alone does not prove that
every
magnetic array is initialized and finite, so I would not rank the EOS and
magnetic possibilities until those checks are available.
Please also report the Einstein Toolkit release and the exact GRHayL and
GRHayLET commit IDs used to build the executable. Older combinations may
predate relevant fixes. For example, GRHayLET pull request 14 fixed missing
error-control flow in the HybridEntropy, Tabulated, and TabulatedEntropy
recovery branches (not the Hybrid branch):
https://github.com/GRHayL/GRHayLET/pull/14
Finally, the current Gallery `bns.par` uses GRHydro with an ideal-fluid EOS,
not IllinoisGRMHD with an SFHo table. Changing the hydrodynamics thorn and
EOS
is therefore a substantial new configuration rather than a small change to a
validated Gallery setup:
https://einsteintoolkit.org/gallery/bns/index.html
https://einsteintoolkit.org/gallery/bns/bns.par
The most useful items to send next would be the complete iteration-0 C2P
diagnostic block, the first warning/error and final output lines, the batch
exit status, the copied runtime parameter file, and the build commit IDs.
Best regards,
-Zach
* * *
Zachariah Etienne
Prof. of Physics, U. of Idaho
Adjunct Prof. of Physics & Astronomy, West Virginia U.
https://etienneresearch.com
https://blackholesathome.net
On Fri, Jul 3, 2026 at 3:10 AM MAKINO Taiki via Users <
users at einsteintoolkit.org> wrote:
> Dear Jordan
>
> Thank you for your quick response.
>
> I understand what I should do. I will wait for feedback from the others,
> and I consider attending the weekly meeting.
>
> Thank you again for your advice.
>
> Best regards,
>
> Taiki Makino
> Graduate School of Science, Nagoya University
> *makino*.taiki.g6 at s.mail.nagoya-u.ac.jp
> <makino.taiki.g6 at s.mail.nagoya-u.ac.jp>
>
>
> 取得 Outlook for Mac <https://aka.ms/GetOutlookForMac>
> *差出人: *Jordan Nicoules <jnicoules at ua.pt>
> *日付: *金曜日, 2026年7月3日 18:38
> *宛先: *MAKINO Taiki <makino.taiki.g6 at s.mail.nagoya-u.ac.jp>
> *CC: *users at einsteintoolkit.org <users at einsteintoolkit.org>
> *件名: *Re: Question about IllinoisGRMHD C2P failure and kuibit Psi4
> extraction
>
> Hi Taiki,
>
> (apologies for my previous typo)
>
>
> The next step would be to understand why the simulation crashes and
> doesn't go past t=0. This is beyond what I can quickly hotfix I think, so *I'm
> calling to other members to answer that.* Potentially, if you have the
> chance, the weekly calls can also be an opportunity to interact with people
> directly and put forth your questions.
>
>
> Regarding files that have more than one line of data, that would make
> sense for the 1D profiles, but they still probably only correspond to
> iteration 0 (t=0).
>
>
> Best,
>
>
> Jordan
>
>
>
> ------------------------------
> *From:* MAKINO Taiki <makino.taiki.g6 at s.mail.nagoya-u.ac.jp>
> *Sent:* Friday, July 3, 2026 10:14:57 AM
> *To:* Jordan Nicoules
> *Cc:* users at einsteintoolkit.org
> *Subject:* Re: Question about IllinoisGRMHD C2P failure and kuibit Psi4
> extraction
>
> *CUIDADO:* Email de um sistema externo. Cuidado com links, anexos e
> pedidos de dados/senhas.
> *CAUTION:* Email from an external system. Be careful with links,
> attachments, and requests for data/passwords.
> Dear Jordan
>
> Thank you for your response.
>
> When I checked the result of my simulation, SimDir.gws[100][(2,2)].x was
> indeed 0 as you said and the simulation stopped at iteration=0.
>
>
> Also, I checked the psi4 ASCII files and all of them contained only one
> line. (However, the ASCII files that do not contain information of psi4
> have several lines of data.)
>
>
> I am not sure what is causing this. What should I do next ?
>
>
> Best regards,
>
>
> Taiki Makino
>
> Graduate School of Science, Nagoya University
>
> *makino*.taiki.g6 at s.mail.nagoya-u.ac.jp
> <makino.taiki.g6 at s.mail.nagoya-u.ac.jp>
>
> 取得 Outlook for Mac <https://aka.ms/GetOutlookForMac>
> *差出人: *Jordan Nicoules <jnicoules at ua.pt>
> *日付: *木曜日, 2026年7月2日 19:31
> *宛先: *MAKINO Taiki <makino.taiki.g6 at s.mail.nagoya-u.ac.jp>;
> users at einsteintoolkit.org <users at einsteintoolkit.org>
> *件名: *Re: Question about IllinoisGRMHD C2P failure and kuibit Psi4
> extraction
>
> Hi Taiko,
>
>
> I will (partly) skip the first part of your message and let others reply
> on that, as I don't have experience with NS simulations.
>
>
> Regarding the gravitational wave extraction, I think you are accessing the
> data correctly (your example gives the time series for one mode at one
> radius), and there should be more than one element in general. From your
> parameter file, it seems the output should go to an ascii file (it's the
> default), so you can double check in that file too.
>
> However, since you have `Multipole::out_every = 128`, does your simulation
> reach iteration 128 and later? In particular, because you're getting nans
> at iteration 0, does the simulation actually go past t=0? Given what you're
> saying, I guess your single element of `SimDir.gws[100][(2,2)].x` is 0 (i.e
> t=0).
>
>
> Best,
>
>
> Jordan
>
>
>
> ------------------------------
> *From:* Users <users-bounces at einsteintoolkit.org> on behalf of MAKINO
> Taiki via Users <users at einsteintoolkit.org>
> *Sent:* Thursday, July 2, 2026 12:39:00 AM
> *To:* users at einsteintoolkit.org
> *Subject:* [Users] Question about IllinoisGRMHD C2P failure and kuibit
> Psi4 extraction
>
> *CUIDADO:* Email de um sistema externo. Cuidado com links, anexos e
> pedidos de dados/senhas.
> *CAUTION:* Email from an external system. Be careful with links,
> attachments, and requests for data/passwords.
> Dear Einstein Toolkit Users,
>
> I am a new Einstein Toolkit user, and I have some questions about a BNS
> simulation.
>
> I am trying to run a BNS simulation using bns.par, which I downloaded from
> Einstein Toolkit Gallery (
> https://einsteintoolkit.org/gallery/bns/index.html) and slightly modified.
>
> I can successfully create and submit the simulation but the following
> message appears during the simulation:
>
> INFO (IllinoisGRMHD): C2P: Iter. # 0, Lev: 5 NumPts= 132300 | Backups: 0 0
> 0 | Fixes: VL= 0 rho*= 0 Averaged pts = 38250 Font1D 38250 | Failures: 2250
> InHoriz= 0 / 0 | 1.00 iters/gridpt Error, Sum: rho 6.378e-08, 2.433e-05 |
> tau nan, -nan | Sx nan, nan | Sy nan, nan | Sz nan, nan
>
> To investigate this problem, I added the following parameters:
> Multipole::interpolator_name = "Lagrange polynomial interpolation"
> Multipole::interpolator_pars = "order=4"
>
> I also changed the parameter Driver::ghost_size from 3 to 4, but the same
> error occurred.
>
> I would like to ask:
>
> - What could cause tau, Sx, Sy, and Sz to become NaN in this situation?
> - Does this indicate a failure of C2P conversion?
> - Are there any parameter settings or initial data that I should check
> first?
>
>
> Additionally, I have another question. When I use kuibit to extract Psi4,
> I expected to obtain the full time series (or all available modes) but the
> returned object contains only one element, for example,
> SimDir.gws[100][(2,2)].y=[-9.79586237e-15-5.27476073e-14j].
>
> Other attributes such as
> SimDir.gws[100][(2,2)].x
>
> also contain only one element. Is this a correct behavior or could it
> indicate a problem with my .par file?
>
> For reference, I have attached my parameter file.
>
> Thank you very much for your time and any suggestions.
>
> Best regards,
>
> Taiki Makino
>
> Graduate School of Science, Nagoya University
>
> *makino*.taiki.g6 at s.mail.nagoya-u.ac.jp
> <makino.taiki.g6 at s.mail.nagoya-u.ac.jp>
>
> 取得 Outlook for Mac <https://aka.ms/GetOutlookForMac>
>
>
> 取得 Outlook for Mac <https://aka.ms/GetOutlookForMac>
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20260723/c9b1d5ed/attachment-0001.htm>
More information about the Users
mailing list