Users July 2026

users@lists.einsteintoolkit.org

11 participants
24 discussions

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 20 Jul '26

20 Jul '26

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Issues configuring SimFactory for PBS cluster (Hubble) - Jobs failing to submit correctly
by MAKINO Taiki 17 Jul '26

17 Jul '26

Dear Einstein Toolkit Users, I am trying to configure SimFactory to submit jobs using PBS on a machine named "Hubble". Initially, the automatically generated hubble.ini was configured for a local login node submission. I attempted to modify the submission commands to support our PBS environment, but the jobs are not running properly. Here are the modifications I made to the .ini configuration: * submit: Changed from: exec nohup @SCRIPTFILE@ < /dev/null > @RUNDIR@/@SIMULATION_NAME@.out 2> @RUNDIR@/@SIMULATION_NAME@.err & echo $! To: qsub @SCRIPTFILE@ * stop: Changed from: pkill -g $(ps -o pgid= -p @JOB_ID@) To: qdel @JOB_ID@ * getstatus: Changed from: ps @JOB_ID@ To: qstat -f @JOB_ID@ Symptoms: 1. When running simfactory/bin/sim submit, no .out or .err files are generated at all. 2. When running simfactory/bin/sim create-submit, the .out and .err files are generated, but the .out file only contains the PBS job ID, and the .err file is completely empty. 3. No simulation output files or directories are created inside the designated simulation directory. As you can see from the symptoms, create-submit successfully captures the job ID, but the simulation does not run and no output is generated. Could anyone guide me on the correct way to write the submit option or the template script for PBS in SimFactory? Any working examples of a PBS configuration (.ini and .sub files) would be greatly appreciated. For your reference, I have attached my .ini file and .sub file that I am currently using. Thank you very much for your help. Best regards, Taiki Makino Graduate School of Science, Nagoya University makino.taiki.g6(a)s.mail.nagoya-u.ac.jp<mailto:makino.taiki.g6@s.mail.nagoya-u.ac.jp> 取得 Outlook for Mac<https://aka.ms/GetOutlookForMac>

1 0

Meeting minutes for 2026-07-16
by Deborah Ferguson 16 Jul '26

16 Jul '26

Present: Roland, Deborah, Steve, Lucas, Peter, Beyhan, Maxwell, Jordan, Noora, Yosef, Zach Chair: Peter Minute taker: Deborah Minutes: New ETK release done! Kuibit release wrong on download page will be fixed Bitbucket issues Zach been working on figuring out what exactly needs to be done to convert over to github with the help of clankers has simplified the process greatly into just a few steps some trickiness about mapping bitbucket users to github users maybe just list the bitbucket user maybe just email address if that’s possible clankers didn’t have a plan for handling users yet was just going to do names in plain text Zach has changed this to make a user map between bitbucket and github users Accumulated updates after the release CarpetX / AMReX there will be a breaking change to CarpetX which we wanted to do early in the release cycle McLachlan updates being brought back into focus Mailing list questions Jordan about using GetComponents can it be used to update everything to the newest release? Roland: the script doesn’t change branches so GetComponents can’t switch you to a new release typically just create a new Cactus directory with the new release Steve: it’s a feature we could add, in principle, if there is interest Jordan about bug in CarpetRegrid2 has opened a ticket Question about IllinoisGRMHD C2P failure and Psi4 extraction Psi4 aspect has been answered still waiting for help about IllinoisGRMHD part it’s crashing immediately getting nans Zach: in IllinoisGRMHD itself, where that error message appears, give some helpful tips about how to go about debugging turn on poisoning likely an issue of feeding in some data but not all the data that illinois GRMHD needs will reply to the email Open tickets Test Derivs fails on stampede with NaNs Steve reproduced the bug but with the help of clankers may have found a solution Needs more eyes Split the ET right now we have a giant thornlist with basically two sets of thorns (Carpet and CarpetX) could split up the Carpet and the CarpetX thornlists could help with code duplication comes with some tradeoffs Noora encountering an issue with compiling when compiling, it was pulling the Carpet versions instead of the CarpetX versions some of the thorns aren’t part of the current ETK release Maxwell what about splitting them but keeping the name differences? Steve if we get rid of the X in some names, then things like TwoPunctures don’t have to be duplicated Lucas which thorns can be made compatible across drivers? twopunctures seems like an exception at some point, we’re probably going to have a breaking change Roland we can make some level of compatibility with just a little bit of performance loss Steve went through with the help of AI to see what thorns would be easy to make compatible with both Yosef development and code maintenance is mostly a volunteer effort deprecate Carpet once CarpetX can replicate all science proposes that long term compatibility doesn’t need to be an overarching goal Steve agrees that we will have to move on and abandon code eventually needs further discussion Zach solving many of these issues could be done by formalizing Roland’s tool that creates a thornlist from the parameter files more informative error messages would be useful as well Jordan can someone explain the difference between Carpet and CarpetX? Lucas and Zach: it’s very nontrivial. loops are different for example. ccl files differ a bit Zach: much easier with clankers Yosef which thorns are actually compatible with both Steve: shared a markdown file with that Roland: only a few things, like Time his impression is that there’s just a handful of thorns that are the same and most things are different treats them as entirely different codes Peter what’s the timescale for actually getting CarpetX to the point of replacing Carpet for production runs let’s discuss this further in future meetings AOB Steve made a few updates to the ETK tutorial server should run more smoothly now ----------------------------------------- Deborah Ferguson Assistant Professor The University of Rhode Island

1 0

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 15 Jul '26

15 Jul '26

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 13 Jul '26

13 Jul '26

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

The Thirty-First Release of the Einstein Toolkit
by Steven Brandt 10 Jul '26

10 Jul '26

Release Announcement Click here to read the announcement in HTML (with hyperlinks): https://einsteintoolkit.org/about/releases/ET_2026_05_announcement.html We are pleased to announce the thirty-first release (code name "Hypatia") of the Einstein Toolkit, an open-source, community-developed software infrastructure for relativistic astrophysics. The major changes in this release include: New thorns in this release: * Cottonmouth (CottonmouthBSSNOK4m, CottonmouthZ4c4m, CottonmouthGaugeWaveID, CottonmouthLinearWaveID) -- A new suite of GPU-accelerated spacetime evolution thorns for CarpetX, generated using the Einstein Engine code generator. CottonmouthBSSNOK4m and CottonmouthZ4c4m implement the BSSNOK and Z4c formulations of Einstein's equations with 4th-order stencils and matter support; CottonmouthGaugeWaveID and CottonmouthLinearWaveID provide corresponding test initial data. * CanudaX (CanudaX_BSSNMoL, CanudaX_NPScalars, CanudaX_ExactID) -- A new GPU-accelerated BSSN spacetime evolution suite for CarpetX, based on the Canuda library. Includes a BSSN evolution thorn, Newman-Penrose scalar Psi4 extraction, and exact initial data (Kerr, Teukolsky waves, Apples-with-Apples test cases). * TwoPuncturesX -- A new thorn providing a C++ interface to TwoPunctures binary black hole initial data, for use with the CarpetX driver. * ET_BHaHAHA -- A new thorn providing an Einstein Toolkit interface to BHaHAHA, the BlackHoles@Home Apparent Horizon Algorithm. It finds and tracks apparent horizons in numerical relativity simulations by interpolating ADMBase metric data to the spherical grids used by the BHaHAHA library, running the hyperbolic-relaxation horizon finder, and writing AHFinderDirect-style diagnostics. Along with a new gallery example: * Black hole-neutron star merger: simulation of a black hole-neutron star merger using a simple equation of state and parameters targeted to match GW230529 Updated thorns and components: * CarpetX -- Added support for integer and complex grid array and scalar types; custom OpenMP reductions for NVIDIA HPC SDK compiler compatibility; named OpenMP critical sections; pinned Particles in interpolation. * GRHayL -- Added Noble2D con2prim for tabulated equations of state; generalized and cleaned con2prim backup strategy; new tabulated EOS reader; Doxygen documentation added. * GRHayLHD/GRHayLHDX -- Updated to use new tabulated EOS reader and new GRHayL API; fixed schedule to correctly use volume integrals group. * TOVola -- Bug fixes for Lagrange interpolation stability; regression tests made surface-aware; updated for new GRHayL API. * AHFinderDirect -- Fixed regex patterns in param.ccl for improved portability. * TwoPunctures -- Refactored to C++ with namespace isolation, enabling the new TwoPuncturesX thorn; added W-type precollapsed lapse option. * NewRadX -- Added support for CapyrX multipatch grids. * QuasiLocalMeasures -- Corrected sign in ADM linear momentum calculation. * SummationByParts -- Fixed duplicated switch construction and 4th-order second derivative operator scheduling. * CCE_Export -- Added configurable array size parameter; minor documentation and portability fixes. * POWER -- Fixed detection of smallest timestep across radii; cleaned up output formatting. * Llama -- Fixed inconsistent volume calculation. * LeanBSSNMoL -- Removed explicit Cray pointer variable declarations for improved portability. * SphericalSurface -- Scheduling fix to ensure Set runs after Setup. * Kuibit -- Bumped to 1.6.1. Updated external libraries: * AMReX -- Updated to version 25.11; added configurable CUDA architecture selection. * ADIOS2 -- Added missing stdint.h include for compatibility with GCC 15; improved MPI library handling. * Boost -- Added new external library thorn for Boost 1.84.0. * GSL -- Updated to GSL 2.8. * HDF5 -- Improved system library path detection. * PETSc -- Updated to PETSc 3.23.4. * Silo -- Build fix: disabled config-site directory reading. In addition, bug fixes accumulated since the previous release have been included, as well as stability and optimization improvements through Simfactory. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems. It builds on numerous software efforts in the numerical relativity community, including codes to compute initial data parameters, the spacetime evolution codes Baikal, lean_public, McLachlan, CanudaX, and Cottonmouth, analysis codes to compute horizon characteristics and gravitational waves, and the relativistic (magneto)hydrodynamics codes GRHayLHD, GRHayLHDX, GRHydro, and IllinoisGRMHD. Data analysis and post-processing are handled by the kuibit library. The Einstein Toolkit also contains a 1D self-force code. The toolkit supports two AMR drivers: the established Carpet infrastructure, and the newer CarpetX infrastructure built on the AMReX framework, which adds support for GPU acceleration and block-structured adaptive mesh refinement on modern supercomputing architectures. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure, providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The Einstein Toolkit uses a distributed software model. Its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers ensure quality control for modules included in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve staff and faculty from five different institutions and host weekly meetings that are open to anyone. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought-out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org, or contact the users mailing list users(a)einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 2411068 (E=mc2: Enabling Multimessenger astrophysics through Community-driven Cyberinfrastructure) and NSF 2004157/2004044/2004311/2004879/2003893/2114582/2227105 (Enabling fundamental research in the era of multi-messenger astrophysics). The Einstein Toolkit contains about 400 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. Contributors Among the many contributors to the Einstein Toolkit and to this release in particular, important contributions to new and existing components were made by the following authors: * Benjamin Meyers * Beyhan Karakas * Bruno Giacomazzo * Cheng-Hsin Cheng * Chloe B. Richards * David Boyer * Erik Schnetter * Gabriele Bozzola * Jordan Nicoules * Leonardo Werneck * Liwei Ji * Lucas Timotheo Sanches * Marco Brito * Maxwell Rizzo * Max Morris * Peter Diener * Rahime Matur * Roland Haas * Samuel Cupp * Samuel Tootle * Steven R. Brandt * Zachariah Etienne How to upgrade from Martin D. Kruskal Release (ET_2025_05) To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download.html) on the Einstein Toolkit website for download instructions. The SelfForce-1D code uses a single git repository; thus, using git pull; git checkout ET_2026_05 will update the code. To install Kuibit, do the following: pip install --user -U kuibit==1.6.1 Machine notes Supported (tested) machines include: * Debian, Ubuntu, Fedora, Mint, OpenSUSE, Alma, and Rocky installations with dependencies installed as prescribed in the official installation instructions * Anvil * Deep Bayou * Delta * DIRAC/Cosma 8 * Expanse * Frontera * Queen Bee 3 and 4 * Sunrise * Supermike Note for individual machines: * TACC machines: defs.local.ini needs to have `sourcebasedir = $WORK` and `basedir = $SCRATCH/simulations` configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. All repositories participating in this release carry a branch ET_2026_05 marking this release. These release branches will be updated if severe errors are found. The "Hypatia" Release Team on behalf of the Einstein Toolkit Consortium (2026-07-10) * Steven R. Brandt (release manager) * Beyhan Karakas (assistant release manager) * Roland Haas (assistant co-chair) * Bing-Jyun Tsao * Deborah Ferguson * Cheng-Hsin Cheng * Leonardo Werneck * Lucas Timotheo Sanches * Maxwell Rizzo * Noora Ghadiri * Peter Diener * Rahime Matur * Yuvraj Sharma July 10, 2026

1 0

meeting minutes for 2026-07-09
by Steven Brandt 09 Jul '26

09 Jul '26

# ET meeting July 9, 2026 ## Present: Steve, Roland, Lucas, Zach, Peter, Beyhan, Jordan, Maxwell Rizzo, Makino Taiki ## Release will be pushed out tomorrow - https://einsteintoolkit.org/about/releases/ET_2026_05.md - https://sandbox.zenodo.org/records/536114 - Deborah and Yurov could not make the meeting but the gallery example is working. - Tests are mostly passing. - Jay was missing from the contributors. Is that an error? - Need to contact Noora for her gallery example. ## Migration of ticket system to Bitbucket. Zach has volunteered. - Needs to happen before August 20 - Need to update the website to point us at the new place. - We need a mechanism to close the Bitbucket side to avoid confusion. - If we delete the repo we can get a forwarding address. - A repo under einsteintoolkit called tickets. - Can disable pull requests for repos. - Mapping ids from bitbucket to another name. - Bitbucket can export a json file with all the tickets. Need to retain the ticket numbering. ## Mailing List - Jordan: CarpetRegrid2 does not catch change in radius of the levels when you set the grid scalar. Seeing different output depending on how often he does output. He should open a ticket. - Helvi: Requesting info for reporting. - Illinois GRMHD Con2prim failures. ## Other issues - Maxwell: Can't trigger of algebra: CCTK_EvaluateFormula. Center of mass can drift from the origin due to gauge condition and truncation error. Maybe discuss next week. "No matter how hard you try to teach your cat general relativity, you're going to fail." -Brian Greene.

1 0

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 08 Jul '26

08 Jul '26

1 0

Bug in CarpetRegrid2 (revealing a deeper issue?)
by Jordan Nicoules 07 Jul '26

07 Jul '26

Dear all, A couple of weeks ago, I found out that there was a bug in CarpetRegrid2. Namely, it does not properly check for a change of radius of the refinement levels (when they are steered by a Trigger for instance). Looking a bit closer, it seems that the computed `drfac` in `CarpetRegrid2/src/regrid.cc` is nan, and thus `do_recompose` is not set to true. In particular, it means it is possible to change the radii if something else triggers a regridding (for instance, changing the number of active levels). However, while I was trying to understand and fix this bug, it appeared there might be a deeper issue. I'm struggling to understand if it may be related to memory, compilation, ... It would be helpful if any of you had some insight to share, or could try to replicate the issue. To that effect, I'm attaching a minimal parfile adapted from CarpetRegrid2/test/regrid2_granularity.par, as well as a modified regrid.cc file that should allow you to reproduce the situation. More details on the situation below. Best, Jordan ---------------------------- ----- More details ----- ---------------------------- To figure out why drfac is nan, I added manual debugging information prints in the code. Depending on what information I ask, and in what order, I don't get consistent nor sensible results. I'm reaching out here first instead of opening a ticket because I can't pinpoint the precise issue. I haven't tried different compilers (I have gcc 11.4.0 locally), but this is happening even with a debug build (see attached make.config.defn file). More precisely, printing `rad`, `oldrad`, `rad - oldrad` is fine (done through std::cout). Printing (CCTK_VINFO) the numerator and denominator of `drfac` separately gives the right values for them, but the quotient `drfac` is wrong (it is 0, and not nan anymore, even when it shouldn't). If I create new local variables ``` CCTK_REAL const num = sqrt(sum(ipow(rad - oldrad, 2))); CCTK_REAL const den = sqrt(sum(ipow(oldrad, 2))); ``` and print them, they both contain the same value. It looks like the values are as if it uses rad=0 and whatever oldrad used to have before (which can be thus changed, depending on the order of the print lines). Finally, if I do print a lot of other info (in particular the intermediate computation steps), it seems it finally gets drfac right. I had a look at CarpetLib's vect.hh and defs.hh too, but I can't seem to understand what is going on. Note that there might be another small bug in the thorn, but fixing that didn't impact what I'm describing. I'm just attaching the other modified files (schedule.ccl and initialise.cc) for completeness, as it may lead to a fix eventually.

1 0

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 06 Jul '26

06 Jul '26

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users July 2026