Dear Taiki,

This is an experimental response prepared using a set of LLM agents and as a
test of the newly created GRHayLET knowledge base:
https://github.com/GRHayL/GRHayLET/blob/main/AGENTS.md

Zach will be traveling for the next month and hopes that this response will
be helpful in the meantime.

The single Psi4 value is consistent with the simulation stopping at iteration
0. In kuibit, `SimDir.gws[100]` selects the extraction radius r=100, and
`[(2,2)]` selects one spherical-harmonic mode. The resulting `.x` and `.y`
arrays are the time and complex data samples for that particular radius and
mode; they are not containers for all radii or modes. You can inspect what was
read with

    print(SimDir.gws.radii)
    print(SimDir.gws[100].available_lm)
    print(SimDir.gws[100][(2,2)].x)

Since the last command returns only t=0 and the Psi4 ASCII files contain only
one row, kuibit appears to be reading the available output correctly. With
`Multipole::out_every = 128`, the next sample would be written at iteration
128; the attached parameter file instead uses 64, so its next sample would be
at iteration 64. Neither setting can produce another sample if the evolution
never advances past iteration 0. Files containing spatial profiles can have
many rows while still representing only iteration 0.

The IllinoisGRMHD message does report C2P recovery failures, but it is an INFO
message and does not by itself explain why Cactus stopped:

* `Averaged pts = 38250 Font1D 38250` says that 38,250 points first required
  neighbor-averaging attempts and then reached the Font1D fallback.
* `Failures: 2250` says that recovery still failed at 2,250 points and those
  points were reset to atmosphere.
* These are counters for the local grid data handled by that printed C2P
  invocation, not necessarily global counts over the whole simulation.
* `tau nan` and the NaN momentum diagnostics mean that an aggregate
  post-recovery error calculation encountered non-finite values. They do not
  imply that every value of tau or every momentum component is NaN.

Thus, there are genuine C2P failures, but C2P may be reacting to invalid
metric, primitive, EOS, or conservative input rather than creating the first
NaN. The first priority is to distinguish the reason for termination from the
first source of non-finite data.

First, please inspect the batch-system exit status and the final 100--200 lines
of both stdout and stderr. Search backward for the termination request,
especially `WARNING level 0`, `CCTK_ERROR`, NaNChecker, TerminationTrigger,
MPI abort, or scheduler kill messages; an unrelated earlier warning need not
explain the stop. The attached parameter file also enables
`TerminationTrigger::termination_from_file` with the file name `TERMINATE`.
Please check for a stale `TERMINATE` file in the run/output directory. Such a
file can request a clean stop at iteration 0 even if the C2P INFO message did
not terminate the run.

There also appears to be a mismatch between the quoted diagnostic and the
requested recovery configuration. If the quoted text is the complete
diagnostic block, a line containing `Font1D` but neither entropy nor Y_e
resembles the Hybrid non-entropy path in current IllinoisGRMHD. In contrast,
the attached file requests a tabulated EOS with entropy evolution and three
backup routines. The TabulatedEntropy diagnostic includes entropy and Y_e and
does not print that `Font1D` field:

https://github.com/GRHayL/GRHayLET/blob/301648c62bb87544adc7399d3239fef97ae0e5d4/IllinoisGRMHD/src/Hybrid/conservs_to_prims.c#L334-L354

https://github.com/GRHayL/GRHayLET/blob/301648c62bb87544adc7399d3239fef97ae0e5d4/IllinoisGRMHD/src/TabulatedEntropy/conservs_to_prims.c#L337-L361

Please check the complete diagnostic block, the startup schedule, and the
parameter file copied into the simulation output directory. In particular:

1. Run once in a fresh output directory with `IOUtil::recover = "no"`, or
   otherwise disable recovery explicitly, and ensure that no stale termination
   file is present. A fresh output directory alone is insufficient here
   because the attached `IOUtil::recover_dir` names a separate
   `output_directory_01/checkpoints` directory.
2. Confirm that the runtime parameter file contains

       GRHayLib::EOS_type = "Tabulated"
       GRHayLib::evolve_entropy = yes
       GRHayLib::con2prim_routine = "Palenzuela1D"

3. Check whether `ID_converter_ILGRMHD` is active and whether startup prints
   `Running GRHayLib with backward-compatible settings`. In current
   IllinoisGRMHD, activating that thorn adds legacy Hybrid scheduling and
   initializes Noble2D/Font1D compatibility settings independently of the
   modern EOS-selected schedule, potentially creating conflicting paths. For
   a modern GRHayLib/GRHayLID setup, remove it and confirm that only the
   TabulatedEntropy C2P routine is scheduled. The attached file does not list
   it, making this check especially useful for detecting submission of a
   different generated parameter file or executable.
4. Keep `Carpet::ghost_size = 3`, as already specified in the attached file.
   IllinoisGRMHD requires at least three ghost zones, and increasing this to
   four will not repair invalid C2P input.
5. Leave the Multipole interpolation settings aside until the evolution
   advances. Multipole analysis cannot repair an iteration-0 Prim2Con/C2P
   problem.

For a more direct NaN check, add `NaNChecker` to `ActiveThorns` and use
something like

    NaNChecker::check_after     = 0
    NaNChecker::check_every     = 1
    NaNChecker::check_for       = "both"
    NaNChecker::action_if_found = "just warn"
    NaNChecker::verbose         = "all"
    NaNChecker::check_vars      = "
      ADMBase::metric ADMBase::lapse ADMBase::shift
      HydroBase::rho HydroBase::press HydroBase::eps
      HydroBase::entropy HydroBase::Y_e HydroBase::temperature
      IllinoisGRMHD::grmhd_velocities
      IllinoisGRMHD::u0
      IllinoisGRMHD::Ax IllinoisGRMHD::Ay IllinoisGRMHD::Az
      IllinoisGRMHD::phitilde
      IllinoisGRMHD::grmhd_B_stagger
      IllinoisGRMHD::grmhd_B_center
      IllinoisGRMHD::grmhd_conservatives
      IllinoisGRMHD::ent_star IllinoisGRMHD::Ye_star
    "

NaNChecker reports fields at its scheduled check point; that timing depends on
the NaNChecker version and does not prove which routine first produced a
non-finite value. To distinguish invalid initial primitives from Prim2Con
output and C2P effects, check immediately before Prim2Con, immediately after
Prim2Con but before C2P, and immediately after C2P. The NaNChecker
documentation is:

https://einsteintoolkit.org/thornguide/CactusUtils/NaNChecker/documentation.html

A finite rho diagnostic together with non-finite tau and momentum only weakly
narrows the cause. GRHayL computes the conserved density from fewer inputs,
whereas conserved energy and momentum use additional thermodynamic, velocity,
metric, and magnetic quantities:

https://github.com/GRHayL/GRHayL/blob/8ef7e60884360ed4e6bef52c1be8fcff4558f3f5/GRHayL/Con2Prim/compute_conservs.c#L92-L98

I would therefore check lapse and metric components, velocity/Lorentz factor,
pressure, specific internal energy, temperature, Y_e, and vector-potential or
magnetic-field components at iteration 0. Although the attached file sets
`Seed_Magnetic_Fields::A_b = 0`, that setting alone does not prove that every
magnetic array is initialized and finite, so I would not rank the EOS and
magnetic possibilities until those checks are available.

Please also report the Einstein Toolkit release and the exact GRHayL and
GRHayLET commit IDs used to build the executable. Older combinations may
predate relevant fixes. For example, GRHayLET pull request 14 fixed missing
error-control flow in the HybridEntropy, Tabulated, and TabulatedEntropy
recovery branches (not the Hybrid branch):

https://github.com/GRHayL/GRHayLET/pull/14

Finally, the current Gallery `bns.par` uses GRHydro with an ideal-fluid EOS,
not IllinoisGRMHD with an SFHo table. Changing the hydrodynamics thorn and EOS
is therefore a substantial new configuration rather than a small change to a
validated Gallery setup:

https://einsteintoolkit.org/gallery/bns/index.html

https://einsteintoolkit.org/gallery/bns/bns.par

The most useful items to send next would be the complete iteration-0 C2P
diagnostic block, the first warning/error and final output lines, the batch
exit status, the copied runtime parameter file, and the build commit IDs.

Best regards,