[Users] Meeting minutes for 2025-01-16

[Lucas Timotheo Sanches]

Present: Lucas (Minutes), Steven R Brandt, Johnny Tsao, Maxwell Rizzo, Roland
Hass (Chair), Erik Schnetter.

# Vote to change the cadence of Einstein Toolkit releases.
  * After intense discussions it was decided (by the voting of those who were
    present) that the releases shall now be **every 12 months, every May**
  * It was agreed that this decision may be revisited in the future, in case
    the community finds it necessary.

# Discussions on minor releases and continuous integration
  * The change of cadence prompted intense discussions on whether we should
    do point releases with important bug fixes in between releases. Several
    points were proposed, in particular the following ideas were proposed
    1. Do not perform point releases, as this would be equivalent to having
       two releases a year.
    2. Leave the decision to do point releases to the current release manager.
    3. Adopt a CI (continuous integration) scheme that would allow us to
       know if any of the changes on the master branch are breaking in the
       machines that we support.
    4. In order to ameliorate difficulties of running tests (even if automated)
       on certain machines, it was proposed that we adopt a system of champions
       and tiers. Each machine would have a champion, responsible for making
       sure that the ETK works correctly on that system. These would be Tier 1
       machines, with full support and working guarantees. Other machines,
       where champions were out of touch or unable to get the toolkit working
       in time, would be called Tier 2 machines.
  * Erik suggested that we postpone this discussion for the next meetings, as
    the topic is broad and requires more thought. All members agreed.

# Reproducibility issues with BNS
  * Wolfgang Tichy has obtained different results with different MPI ranks on
    BNS runs and reported them to Roland. Roland also did a bit of testing with
    the BNS gallery example, writing checkpoints at iteration 0 and 1. He sees
    differences between runs with 4 and 8 MPI ranks.
    Zach reminded Roland that Newton's method exhibits chaotic behavior, which
    his group observed on Illinois GMHD and may explain the results.
    Erik countered the argument saying  we should assume that a bug is more
    likely. Roland says he needs to perform more tests, but he suspects that
    boundary condition interactions may be causing the problems.
    Zach suggested that Roland determines if the hydro quantities change before
    the metric changes, as hydro changes would take some time to propagate to
    metric components. Erik suggested checking different time levels.
    Erik also suggests comparing RHSs. Finally, Zach recommends trying
running it
    through Valgrind, to check accidental memory issues. Roland took note of
    all suggestions and will implement them.

# Upcoming Einstein Toolkit Release
  * NewRadX: Even though the code is completed, it still needs reviews and
    patches to be approved.
  * Z4C: Pending review by reviewers.
  * Cauchy characteristic extraction for Spectre CCE: Deborah Ferguson, who is
    the champion of this code, says that work on it is on her agenda but she is
    currently busy finishing a paper for her group.
  * Cosmology codes: Roland has reached out to Hayley Macpherson to see if she
    has any code that she would like to include. She has not reached back yet.
  * Gallery examples for more modules in the ETK: Roland reports that there are
    two such examples in the works based on Canuda for evolving the Einstein +
    various fields systems. Roland reports summer student projects for new
    gallery examples may not happen as the summer student budget was cut.

# Unanswered question on the mailing list

No questions.

# Open tickets sorted by update time

2845: Small change for fixing C++ namespaces.
2837: Help request with not much progress since last week.

# Tickets ready for review
  * Roland requests for reviewers, particularly those interested in CarpetX, to
    help on the review of the first few tickets such as 2845, which are
    relatively small and easy to review.