Users August 2021

users@lists.einsteintoolkit.org

21 participants
31 discussions

ET and recent Intel compilers
by Hee Il Kim 21 Oct '21

21 Oct '21

Hi, I've encountered various issues from recent Intel compilers. Except for some versions having header file issues, I could manage to build ET executables but actual run stalled at various steps depending on the compiler versions. For example, Case 1. Before initial data generation ... INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 Case 2. During reading Lorene data (e.g., while reading resu.d) Case 3. Evolution started with normal IDs but NaNs produced from the next evolution step. I've encountered the issues in various combinations of OneAPI/IntelMPI and OneAPI/OpenMPI on Centos8 (gcc-8.3.1) machines. There had been no issues for OpenMPI/GCC. But I've just found Case 1 is produced on an old machine with devtoolset gcc-6.3.1/openmpi-1.8.4. You have any hints for the issue? Thanks in advance. Hee Il

2 4

Code failure apparently due to changes in Reduction.c
by Yosef Zlochower 13 Sep '21

13 Sep '21

Hi, A code that we are using seems to no longer work with Cactus. The issue seems to be that the code called CCTK_LocalArrayReductionHandle. The code, as checked out, produces errors because the appropriate handle isn't found. If I modify the function as below, I no longer get the error message int CCTK_LocalArrayReductionHandle(const char *reduction) { int handle; handle = Util_GetHandle(LocalArrayReductionOperators, reduction, NULL); if (handle < 0) { CCTK_VWarn(1,__LINE__,__FILE__,"Cactus", "CCTK_LocalArrayReductionHandle: No handle: '%d' found for reduction operator " "'%s'", handle, reduction); } handle += ARRAY_OPERATOR_HANDLE_OFFSET; //// MY changes return handle; } Should we no longer use CCTK_LocalArrayReductionHandle? -- Dr. Yosef Zlochower Center for Computational Relativity and Gravitation Associate Professor School of Mathematical Sciences Rochester Institute of Technology 85 Lomb Memorial Drive Rochester, NY 14623 Office:74-2067 Phone: +1 585-475-6103 yosef(a)astro.rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

3 7

Einstein Toolkit and modern AMD supercomputer
by Gabriele Bozzola 30 Aug '21

30 Aug '21

Hello, Last week I opened a PR to add the configuration files for Expanse to simfactory. Expanse is an example of the new generation of AMD supercomputers. Others are Anvil, one of the other new XSEDE machines, or Puma, the newest cluster at The University of Arizona. I have some experience with Puma and Expanse and I would like to share some thoughts, some of which come from interacting with the admins of Expanse. The problem is that I am finding terrible multi-node performance on both these machines, and I don't know if this will be a common thread among new AMD clusters. These supercomputers have similar characteristics. First, they have very high cores/node count (typically 128/node) but low memory per core (typically 2 GB / core). In these conditions, it is very easy to have a job killed by the OOM daemon. My suspicion is that it is rank 0 that goes out of memory, and the entire run is aborted. Second, depending on the MPI implementation, MPI collective operations can be extremely expensive. I was told that the best implementation is mvapich 2.3.6 (at the moment). This seems to be due to the high core count. I found that the code does not scale well. This is possibly related to the previous point. If your job can fit on a single node, it will run wonderfully. However, when you perform the same simulation on two nodes, the code will actually be slower. This indicates that there's no strong scaling at all from 1 node to 2 (128 to 256 cores, or 32 to 64 MPI ranks). Using mvapich 2.3.6 improves the situation, but it is still faster to use fewer nodes. (My benchmark is a par file I've tested extensively on Frontera) I am working with Expanse's support staff to see what we can do, but I wonder if anyone has had a positive experience with this architecture and has some tips to share. Gabriele

5 6

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 30 Aug '21

30 Aug '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Meeting Minutes 2021-08-26
by Gabella, William E 26 Aug '21

26 Aug '21

Present: Bill G, Helvi W, Steve B, Miguel G, Yosef Z, Roland H, Leonardo RW, Atul K, Peter D, Zach E, Maria H, Gabriele B Chair: Helvi W, Minutes: Bill G * SPEC benchmark contribution [https://bitbucket.org/einsteintoolkit/tickets/issues/2470 task] [RH, SB] Roland, no real updates, do need to send an extended description and ask if they need more details. Steve would like to know why they asked for this detailed description. https://www.spec.org/cpu2017/Docs/benchmarks/507.cactuBSSN_r.html * next ET release manager [RH] Next release is in November; recall feature freeze is about two weeks before release and hard freeze a few days before. https://docs.einsteintoolkit.org/et-docs/Release_Details https://docs.einsteintoolkit.org/et-docs/Release_Process Steve agreed to be the manager with help from Yosef. Helvi, it is useful to have more than one person to do this. Steve, Roland left off the conditional that Steve will do it if Roland does this "soon," need not be the next release after. ** New inclusions for next release FLRWSolver/Cosmology by Hayley Macpherson and Helvi will review it. Ticket #2549, https://bitbucket.org/einsteintoolkit/tickets/issues/2549/inlcude-flrwsolve… . Kuibit and possibly RePriMand for inclusion. RePriMand has ticket #2551, https://bitbucket.org/einsteintoolkit/tickets/issues/2551/include-reprimand… . RePriMand does have a thorn and an external library for it. Unlikely to build out of the box...requires a couple of other packages/tools. If it requires external libraries not typically available on clusters, we cannot enable it by default. For example, Peter's 1D Force is a model of a stand-alone version that is not built by default and uses SCons for building. Atul, wanted to volunteer for Kuibit review, and he is unable to review Kuibit this semester. Can do less intensive work for the release though. Yosef intends to try to review Kuibit. Review does not check for correctness, checks for running and documentation. Ticket #2438, https://bitbucket.org/einsteintoolkit/tickets/issues/2538/inclusion-of-kuib… . Yosef, build systems question, can RePriMand run on all clusters. Wolfgang says it is mostly Python and if you have some Pip version maybe can install/build that way. Builds using Meson which uses Ninja and not Make. Gabriele, how will Kuibit is to be included in the ET? In the repo or installed by Pip? Roland, could use it in GetComponents and that gives best exposure and lets different branches be checked out. Kuibit was reviewed in the Journal of Open Science Software...there is a link in the ticket. JOSS paper at https://joss.theoj.org/papers/10.21105/joss.03099 . Roland, If the software is independently hosted, we ask for a branch label with the ET name, and then if their Master branch for that software fails, we use the ET branch of their software. Helvi, has a code in the ET and has one branch for the ET and there is another branch her group uses routinely. Roland discusses a couple of options and how to get the dependencies for Kuibit, for example. Gabriele also uses Tags on the Branches that he can make very easily. He likes distributing the repo because of all the examples. Helvi, maybe include these links in the documentation. ** Gallery Examples Helvi has two students that should do examples. * Any Other Business ** Maria, would like documentation on thorns Multipole and QuasiLocalMeasures and SphericalSurface used to extract the GWs. Why/How do they depend on each other and how to tune the parameters. Helvi, Check the Readme for the thorn, some information there. Roland, WeylScal4 everyhwere in 3D space, and Multipole calculates the projection and spin spherical harmonics. QuasiLocalMeasures is separate, concerned with calculations on horizons, like spin, etc, not concerned with waves but with horizons. There are papers cited there, in the README. An older thorn does the same thing. Some documentation was started but does not go very far. SphericalSurface does not do any analysis, it is a way to describe in the code star-shaped surface. Horizon finder will store the shape in the spherical surface to do the isolated horizon quantities. Waveform extraction does not use the SphericalSurface at all. ** Helvi, there is a new monthly call for the NumRel community started by Nils Fischer; the next call is 13th September. The idea is to have speakers from different code communities, next time SXS/Mitman and Meudon/Oertel are presenting. Structure is to describe the project, here is what we are working, and here is where we have difficulties and ask if there are any ideas in the community. https://github.com/sxs-collaboration/nr-community-call/wiki * [https://www.einsteintoolkit.org/tools/unanswered.php unanswered question on mailing list] Perl script is failing, in many ways. It does not display unanswered entries on the mailing list right now. Roland is aware. * [https://bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new… open tickets sorted by update time] Helvi goes through the tickets. #2553, CPPFLAGS and CXXFLAGS trouble when you use a non-g++ compiler, like using the cuda compiler. Roland has a work around. Only affects Cuda. If doing Cuda this might affect you. #2559, HDF5 library generates 2GB of files and we do not use them and causes trouble in some user quotas. Roland, this is cleanup. #2558, link for Power incorrect, ticket is almost closed, need to wait one more week and then will backport the fix. #2538, inclusion of Kuibit, discussed above. * [https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=open… tickets ready for review] Nothing new or discussed. * Any Other Business Next weeks chair and minute taker: Yosef will chair. Maria will take minutes. -- ============================== William Gabella Research Assistant Professor Department of Physics and Astronomy Nashville, TN USA (o) 615-343-2713

1 0

Fwd: [nr-community-call] Next NR community call: Sep 13th
by Helvi Witek 26 Aug '21

26 Aug '21

Dear all, join us for the NR community call on Sept. 13; see below Best wishes Helvi --------------------------------------------- Dr. Helvi Witek Assistant Professor Department of Physics University of Illinois at Urbana-Champaign 247 Loomis Lab 1110 W Green St Urbana, IL 61801 --------------------------------------------- -------- Forwarded Message -------- Subject: [nr-community-call] Next NR community call: Sep 13th Date: Tue, 24 Aug 2021 11:25:42 +0200 From: Nils Fischer <nils.fischer(a)black-holes.org> To: nr-community-call(a)black-holes.org Dear Numerical Relativists, Our next NR community call will be on *September 13th*. Please note this is a week later than usual to accommodate a public holiday in the US. Our speakers will be *Keefe Mitman (Caltech / SXS)* and *Micaela Oertel (Meudon / CompOSE). *You can find the schedule in the wiki: https://github.com/sxs-collaboration/nr-community-call/wiki <https://urldefense.com/v3/__https://github.com/sxs-collaboration/nr-communi…> - Feel free to circulate this information within the NR community. - Please subscribe to our mailing list <https://urldefense.com/v3/__https://groups.google.com/u/1/a/black-holes.org…> to receive info on the community calls in the future (or send a short message to nr-community-call+subscribe(a)black-holes.org <mailto:nr-community-call+subscribe@black-holes.org> to subscribe) and join our Slack <https://urldefense.com/v3/__https://join.slack.com/t/generalrelativity/shar…> for discussions (channel: #nr-community-call <https://urldefense.com/v3/__https://generalrelativity.slack.com/archives/C0…>). - Please sign up for a slot <https://urldefense.com/v3/__https://docs.google.com/forms/d/e/1FAIpQLSfoCnn…> in the schedule to speak at one of the community calls: https://github.com/sxs-collaboration/nr-community-call/wiki <https://urldefense.com/v3/__https://github.com/sxs-collaboration/nr-communi…> Best, Nils -- You received this message because you are subscribed to the Google Groups "NR community call" group. To unsubscribe from this group and stop receiving emails from it, send an email to nr-community-call+unsubscribe(a)black-holes.org <mailto:nr-community-call+unsubscribe@black-holes.org>.

1 0

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 25 Aug '21

25 Aug '21

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 23 Aug '21

23 Aug '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

BHNS simulation issues - Hamiltonian constraint violations and general problems
by Konrad Topolski 23 Aug '21

23 Aug '21

Hi all I am trying to run a BH-NS simulation using at least one of some (and sometimes more) unofficial thors. That would be Kadath for the initial data and possibly THC for hydro evolution. I'm trying to run the simulation either with GRHydro or with THC. In both cases I believe that the initial data gets imported correctly - I have looked at some simple visuals which I'm attaching to this mail. However, as the simulation proceeds, huge hamiltonian constraint violations occur and e.x. the lapse gets driven to 0 globally. This can also be seen. How can I help this? Is it because the puncture is not tracked correctly? Trying to set up a PunctureTracker at the location of the black hole yields a failure very soon, as the shift at puncture is said to be (nan, nan, nan). I'm sending the .par files, outputs (.out), errors (.err) and visualization (.png). Best regards Konrad

2 5

Meeting minutes for 2021-8-19
by Miguel Gracia 19 Aug '21

19 Aug '21

Present: Peter D., Miguel G., Roland H., Steven B., Zach E., Atul K., Leonardo W., Gabriele B., Yosef Z., William G., Maria H. Hayley M., Helvi W. Chair: Steven B. Minutes: Miguel G. * SPEC benchmark contribution Roland sent email. They needed a description, Roland is working on that https://www.spec.org/cpu2017/Docs/benchmarks/507.cactuBSSN_r.html * Next ET release manager search: Roland sent email to people asking: “are you willing to be the release manager?”. Yosef suggested that there should be a release team with a coordinator more than a release manager in charge of everything. Being the release manager shouldn’t be too difficult because Zach in the last release created detailed notes about the process, also he pointed out some stuff in release list that only Steve or Roland can do like updating ZENODO site. It was decided to wait for one more week if not Steve volunteered to do it if Roland does the next one. * New components in the release: Kuibit: Yosef is going to review it, he will try to start soon by installing it, Gabriele said version 1.3 master branch should be most possible candidate for the release. Kuibit documentation should be followed in order to install it or follow the gallery example link<https://www.einsteintoolkit.org/gallery/bbh/index.html>: Atul is going to find out soon if he will be able to help with the review. RePrimAnd: Roland is going to review RePrimAnd library. The release will include only the library. FLRWSolver: Hayley Macpherson is setting everything up to meet the ET inclusion requirements. Helvi is the reviewer. She will start in the coming weeks. * CarpetX activities update: Roland. CarpetX is peaking up steam, Erick main author. Developers group has growth ( from different institutions) aiming for a Hydro code. The contributor rules/guidelines have been updated. Steven: Does CarpetX work on AMD gpu’s? Roland’s not sure but might be possible. * MWRM21<https://midwest-relativity-meeting-2021.physics.illinois.edu/> at UIUC November 11 - 13, 2021 Thursday to Saturday -Midwest relativity meeting planned to be hybrid (online/in person) , everyone is invited. * Unanswered question on mailing list: -ET and recent Intel compilers (Hee Il Kim) mixing compilers might be part of the issue. Steve is going to reply. Other questions: Maria H: is it possible to have BNS evolution where each of the stars has different EOS?. Roland: No, it is not possible in order to do it, one will need to modify the code. * Open tickets SDSC Expanse machine added to machine list. * Tickets ready to review Next call: August 26th Chair: Helvi W. Minutes: William G — end of the call

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users August 2021