Users May 2021

users@lists.einsteintoolkit.org

16 participants
26 discussions

a problem met in running the sample code "Binary, inspiraling neutron stars forming a hypermassive neutron star" on Jimmy's own work station
by 白济民 10 Jun '21

10 Jun '21

Hello, I met a problem when running the sample code "Binary, inspiraling neutron stars forming a hypermassive neutron star" in ET's gallery on my own work station and I'm looking forward to your help. It aborts unexpectedly after running a few minutes. The end of the Output-error-file reads as follows: cactus_sim: /home/bai/ET/Cactus/configs/sim/build/Carpet/helpers.cc:275: int Carpet::Abort(const cGH*, int): Assertion `0' failed. Rank 0 with PID 73447 received signal 6 Writing backtrace to nsnstohmns/backtrace.0.txt Aborted (core dumped) I also uploaded the entire error file for clearance. I built the ET using 64 processors by using the following command: simfactory/bin/sim build -j64 --thornlist thornlists/nsnstohmns.th and I ran the simulation using 20 processors by using the following command: ./simfactory/bin/sim create-submit bns_merger /home/bai/ET/Cactus/par/nsnstohmns.par 20 24:0:0 Yours sincerely: Jimmy

2 13

Re: [Users] Compiling on DiRAC DIaL
by Elley, Matthew 03 Jun '21

03 Jun '21

Hi Roland, both of your suggestions worked, thanks! Matt ________________________________ From: Roland Haas Sent: Wednesday, May 19, 2021 6:31 PM To: Elley, Matthew Cc: users(a)einsteintoolkit.org Subject: Re: [Users] Compiling on DiRAC DIaL Hello Matt, > Cactus/arrangements/ExternalLibraries/PAPI/src/stats.c(20): catastrophic error: cannot open source file "papi.h" > #include <papi.h> > > I cannot find the header file anywhere (i.e. in ET or /usr/include) > so I am not sure how to proceed. Is PAPI a necessary library for ET > function? If you use PAPI_DIR=BUILD to make Cactus build PAPI itself (which you seem to do) then it will install the created PAPI library to Cactus/configs/ET_dirac/scratch/external/PAPI and the include file would be there. Cactus at that point should set up a include search path to find files there, which you can check in: Cactus/configs/ET_dirac/bindings/Configuration/Capabilities/make.PAPI.defn > Compiling with it excluded from the thornlist gives me > > /usr/bin/ld: cannot find -lsz > make[1]: *** > [/lustre/dirac/home/dc-elle1/ET/Cactus/exe/cactus_ET_dirac] Error 1 > > which I am not sure is connected as I am compiling from scratch. That one I suspect is actually being pulled in by HDF5 which can use the sz library but I have seen systems (Windows using MSYS admittedly) where the sz library was called szip. The actual line that cases the failure is probably this one: https://github.com/EinsteinToolkit/ExternalLibraries-HDF5/blob/abf8dbf78e76… You could try two things to avoid this: 1. build your own HDF5 by setting HDF5_DIR=BUILD 2. find out where libsz.so or libsz.a are located (by using he "module show" command on the HDF5 module or asking the admins) and adding that directory to HDF5_LIB_DIRS Please see https://www.einsteintoolkit.org/thornguide/ExternalLibraries/HDF5/documenta… and https://www.einsteintoolkit.org/usersguide/UsersGuide.html#x1-21000B2.2 for details on these option list variables. Yours, Roland > > Thanks, > > Matt > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 31 May '21

31 May '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

New Einstein Toolkit Release (Lorentz)
by Zach Etienne 31 May '21

31 May '21

Release Announcement Click here to read the announcement in HTML (with hyperlinks): https://einsteintoolkit.org/about/releases/ET_2021_05_announcement.html We are pleased to announce the twenty-second release (code name "Lorentz") of the Einstein Toolkit, an open-source, community developed software infrastructure for relativistic astrophysics. The highlights of this release include: * POWER, a Python package to post-process the data products of simulations to compute the gravitational wave strain at future null infinity. * Simfactory is now fully compatible with Python 2 or 3. In addition, bug fixes accumulated since the previous release in November 2020 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including code to compute initial data parameters, the spacetime evolution codes Baikal, lean_public, and McLachlan, analysis codes to compute horizon characteristics and gravitational waves, the Carpet AMR infrastructure, and the relativistic magneto-hydrodynamics codes GRHydro and IllinoisGRMHD. The Einstein Toolkit also contains a 1D self-force code. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve staff and faculty from five different institutions, and host weekly meetings that are open for anyone to join. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought-out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org, or contact the users mailing list users(a)einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 2004157/2004044/2004311/2004879/2003893 (Enabling fundamental research in the era of multi-messenger astrophysics). The Einstein Toolkit contains about 327 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: Larger changes since last release * Vectors: Provide arithmetic operations that mix vector and scalar arguments * TestLoop: Use OpenMP and OpenMP macros * GetComponents: Add --no-update option * CarpetIOHDF5: Improve assertions infrastructure * TestLoopControl: Add OpenMP code to test * simfactory2: Many machine configurations updated and modernized * VolumeIntegrals_vacuum: Add ADM (mass/momentum/angular momentum) integrands * VolumeIntegrals_vacuum & VolumeIntegrals_GRMHD: Add documentation * ID_converter_ILGRMHD: Fix option that perturbs initial data * Llama: Re-allow interpatch boundary interpolation for all Thornburg04 * Cactus: Add CUDA support, fix various compilation issues on modern compilers, fill in missing documentation * EinsteinAnalysis and CactusBase thorns: Improve and fix documentation Contributors This release includes contributions by Gabriele Bozzola, Steven R. Brandt, Brockton Brendal, Zachariah Etienne, Roland Haas, E. A. Huerta, Daniel Johnson, David Radice, Erik Schnetter, and Leonardo Werneck. How to upgrade from DeWitt-Morette (ET_2020_11) To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download.html) on the Einstein Toolkit website for download instructions. The SelfForce-1D code uses a single git repository, thus using git pull; git checkout ET_2021_05 will update the code. Machine notes Supported (tested) machines include: * Default Debian, Ubuntu, Fedora, CentOS 7, Mint, OpenSUSE and MacOS Catalina (MacPorts) installations * Bluewaters * Comet * Cori * Graham * Frontera * Mike / Shelob * Queen Bee 2 * Queen Bee 3 * Stampede 2 * SuperMUC-NG * Summit * Wheeler Note for individual machines: * TACC machines: defs.local.ini needs to have `sourcebasedir = $WORK` and `basedir = $SCRATCH/simulations` configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * SuperMUC-NG: defs.local.ini needs to have `sourcebasedir = $HOME` and `basedir = $SCRATCH/simulations` configured for this machine. You need to determine $HOME and $SCRATCH by logging in to the machine. All repositories participating in this release carry a branch ET_2021_05 marking this release. These release branches will be updated if severe errors are found. The "Lorentz" Release Team on behalf of the Einstein Toolkit Consortium (2021-05-31) * Zachariah B. Etienne * Roland Haas * Steven R. Brandt * William E. Gabella * Peter Diener * Atul Kedia * Miguel Gracia May, 2021

1 0

Meeting Minutes 2021-05-27
by Gabella, William E 27 May '21

27 May '21

Present: Atul K, Bill G, Steve B, Yosef Z, Karima S, Peter D, Leonardo W, Zach E, Ken S, Maria BH, Miguel G, Roland H Chair: Atul K, Minutes: Bill G * Before agenda items? (like quick help questions, etc) None * SPEC benchmark contribution [https://bitbucket.org/einsteintoolkit/tickets/issues/2470 task] [RH, SB] Steve, Roland asked SPEC how things are going, they outlined what more they want from us. They want to be able adjust workload to keep a 40 core Haswell machine busy for many minutes. They could integrate Cactus fine, adjust config.h for various platforms ...they will do that. When testing is done, will start on step 2 proposal, workload is step 3. Their goal is for Serial max memory 1.8 GB and run for 8 min, for Parallel 40 threads 8 GB and 10 mins. They are asking for advice on adjusting workloads. Nothing is urgent but there are things to be done. See above ticket, #2470. * Release Planning [ZE] (Hendrik) Lorentz is the release name. Zach, It is coming along. Release Announcement draft has been shared among maintainers, https://docs.einsteintoolkit.org/et-docs/ET_2021_05_release_announcement . Simfactory should work with Python 2 or 3, compatible with Python 3 stated in the announcement. List of changes since last release should be double checked by everyone. Steve, Maintainers database exists inside the repo. Zach sees developers.txt, and scripts to turn that into other formats. Developers.py creates HTML, and Zenodo.py will also read developers.txt . Release helpers should go into this file for the current release and taken out for next release. Handling this file should be in the Release Planning, etc. See http://einsteintoolkit.org/developers.html and https://bitbucket.org/einsteintoolkit/www/src/master/developers.txt . Zach has been updating the Release Planning documentation https://docs.einsteintoolkit.org/et-docs/Release_Process . One advice is to word wrap at line 72 for the email formatting, he added an online tool to do that. Branches created tomorrow, so any minor changes have to happen today. Do we use SVN? Steve, maybe. Roland, we do not have any Subversion repos anymore, some checkout function does use them. Atul, Zenodo ticket #2524 should be addressed https://bitbucket.org/einsteintoolkit/tickets/issues/2524/zenodo-page-shows… . Zach, Zenodo may not be updated by May 31st. But the DOI from them is part of the bibtex key we ask people to cite. Nice if there, but not checked out for new release, so can be updated later. Zach, Will edit the main ET page, http://einsteintoolkit.org/index.html . Remove the ICERM announcement, keep North American Workshop, and Release Announcement in the center. If you are part of the release team, gallery example checkers included, double check your names at https://bitbucket.org/einsteintoolkit/www/src/master/developers.txt . ** Gallery Examples [ZE] *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2172/test-binary-black… BBH] [SB] Steve, it runs and finished a bit early, so will run it to the end---ran out of time. Working with Kuibit to generate the plots vs Mathematica. Waveforms and trajectories look fine. *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2175/test-single-stabl… TOV] [PD] Peter, re-ran it last night and still works fine. Will update web pages tomorrow/Friday. *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2174/test-multi-patch-… Multipatch] [BG] Bill, Runs fine with and without Presync. Updated the Bitbucket / web pages, had to remind myself how to upload the tarball, what credentials to use (create a APP key on bitbucket but use my username not SSO email, go figure). *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2176/test-binary-neutr… BNS] [AK], Atul, Done but not finished the Bitbucket updates. *** [https://bitbucket.org/einsteintoolkit/tickets/issues/2173/test-poisson-equa… Poisson equation] [MG] Miguel, finished the test and it was fine. Upload the tarballs and files, but need to update the links on the web page. Atul, We should re-run the examples again with the new Release...in June sometime. ** POWER [https://bitbucket.org/einsteintoolkit/tickets/issues/2413/include-power-cod… status] [RH] Maria, did look over the documentation for it and wrote something on Bitbucket and Roland should check it out. Roland, some items in the ticket #2413 are more ET related and not Power related---will clarify in the documentation. Maria, Power works with BBH but not so easy with BNS. Roland, you can run it with BNS but need to specify more items. Atul, add in documentation how to change the frequency with masses, and about quasi-local measure. Roland, want to avoid too much information for running ET in the Power documentation. ** [https://bitbucket.org/einsteintoolkit/tickets/issues?kind=bug&priority=majo… bugs of severity major or worse] Steve, #2533 is not fixed yet. If two ActiveThorns that is the problem, but need better messaging. https://bitbucket.org/einsteintoolkit/tickets/issues/2533/activethorns-not-… Yosef, #2534 Mol never worked well with Grid Arrays, but had a code that did work and it currently does not. Has not worked with 2020 releases, did in the 2019. Compile issues had missing header and maybe Presync problem. Steve, if we do fix it we need to add a test for this issue/problem so we do not break it again. https://bitbucket.org/einsteintoolkit/tickets/issues/2534/evolution-of-grid… ** all [https://bitbucket.org/einsteintoolkit/tickets/issues?milestone=ET_2021_05&s… open tickets] flagged for the next release #2526 is not a blocker and will not be ready by the release. https://bitbucket.org/einsteintoolkit/tickets/issues/2526/mclachlan-has-inc… #2469 Leonardo, I iwll push hard on this today. https://bitbucket.org/einsteintoolkit/tickets/issues/2469/none-of-the-thorn… #2488 Atul, has a minor tag so not critical right now. Zach, Last commits should be minor and not change functionality and should happen today. Will work with Steve and Roland on the Zenodo, etc. But please read the Release Announcement soon, want to finish it. Bill, we should mark the tickets (which are listed as Tasks and not Tickets) for the gallery examples as resolved when you are through with them. They will be re-opened and re-assigned for the next release. [https://www.einsteintoolkit.org/tools/unanswered.php unanswered question on mailing list] Seem to be handled. [https://bitbucket.org/einsteintoolkit/tickets/issues?status=open&status=new… open tickets sorted by update time] #2530, Roland has some ideas to fix it, not high priority and will not do anything before the release to avoid breaking anything. https://bitbucket.org/einsteintoolkit/tickets/issues/2530/thorn-vectors-fai… [https://bitbucket.org/einsteintoolkit/tickets/issues?status=new&status=open… tickets ready for review] Six tickets most still need reviewers. Please volunteer. #963, Peter, did look at it. It is a patch to an old version of McLachlan that is no longer current. Has to wait until after the release. https://bitbucket.org/einsteintoolkit/tickets/issues/963/improve-mclachlan-… * AOB Chair and Minute taker for next weeks: June 3, chair Steven B, minutes Miguel G June 10, chair Peter D, minutes Atul K 0123456789012345678901234567890123456789012345678901234567890123456789 -- ============================== William Gabella Research Assistant Professor Department of Physics and Astronomy Nashville, TN USA (o) 615-343-2713

1 0

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 26 May '21

26 May '21

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 24 May '21

24 May '21

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Meeting Minutes May 20, 2021 ETK
by Hamilton, Maria 20 May '21

20 May '21

Present: Zach Etienne, Yosef Zlochower, Miguel Garcia, Leonardo Werneck, Ken Sible, Atul Kedia, Steve Brandt, Roland Haas, Peter Diener, Maria Hamilton Chair: Peter Diener Minute taker: Maria Hamilton * pre-Agenda : nothing * SPEC benchmark contribution: Steve no news * Workshop update: * Roland: deadline talks registration June 23 * ETK 2021 Workshop Webpage<https://einsteintoolkit.github.io/et2021uiuc/> * ETK 2021 Workshop Announcement<https://hyperspace.uni-frankfurt.de/2021/05/20/north-american-einsteintoolk…> * Scientific committee - Yosef, tasked to contact speakers, Roland will provide email addresses. * New release main features: * POWER: Roland just pushed a new commit, added documentation, Maria will test. * New release issues: BBH Gallery * Steve reran BBH from Gallery, run on Deep(?) Bayou cluster, code dies with assertion failure on number of threads. Made ticket, reported problem to Erik, who said is not pssibe, his code is perfect! * Zach suggested to try running with previous release, if yes, then revert to it * Steve will try, and see if works, although expects is an issue with the cluster, not teh code * Roland explained that it might come from an misunderstanding between threads and computing nodes, a problem due to hwloc1, hwloc2 conflict. He found a workaround for QueenBee and advised Steve to check the QueenBee optionlist. * Steve mentioned that hyperthreading is disabled for most clusters, while here is enabled and threads per core = 1, thus this might be the issue. * Other gallery examples: * BNS: Atul NSTracker link provided * asks what should be updated, if results identical * Roland advises update everything * Leo: works on documentation for VolumeIntegral and will update in the ETK arrangements. * Maria asks if the way VolumeIntegral is set up to follow star center does depend on mesh refinement and if is there any need for AHFinder or PunctureTracker. Zach explains that it tracks the center of the quantity set up to track, for example the lapse of a BBH system, and it does not depend of regridding. As for AHFinder or PunctureTracker, it can use both. * Zach computed constraint viloation, and set it up to avoid interior of horizon, by specifying radius and center, thus substracting off interiors, and also excising region outside some boundary like when tracking constraint violation * Name for the release ->male -> Lorentz -> no bjection, no vote necessary * Peter spoke up, Lorentz<https://en.wikipedia.org/wiki/Hendrik_Lorentz> it is * Multipach Wave Equaton -> Bill travels, sent update to Zach, bumps regarding visualization, will update/improve web page, to make it easier for next person, will update ticket on Mon * Poisson Equation: -> Miguel updated all results, will replace on the website * Roland confirms that will be no more changes of the code after the freeze * TOV -> Peter ran it, everything looks identical, can't use presink yet, and not uploaded yet. Did not know what was used to generate the plot, will use gnuplot and add script on web page, update source tree, rerun and add updates. * Zack: friendly reminder of crunch time, everything needs to be in place for next week, update all that needs to be updated. He will work on release announcement. * Release Process<https://docs.einsteintoolkit.org/et-docs/Release_Process> and Details<https://docs.einsteintoolkit.org/et-docs/Release_Details> * Roland reminded Zach to contact those who contributed and ask if they are willing to be listed - usually should be listed as authors if they are gallery runners * Zach - will take care of Zenodo link asks if its a new one or updated. Roland explains that it keeps track of versions, but needs token change to "become" Steve to be able to update. * Nothing else on the release * Unanswered mailing list queries: Peter - no unanswered email found * Tickets: * Roland couple of tickets, nothing to worry about * Flang compiler Error #2529<https://bitbucket.org/einsteintoolkit/tickets/issues/2529/flang-error-in-ca…> * Problem with Vectors compipation in gcc8, but we use gcc9 * Bug report about initialized variables, if you want some that don't exist. * Peter Erick found Flang issue * End of Agenda: Miguel reported about Gabriel's pull request for machine file Frontera simfactory2. Roland will look into it to be approved * May 27 meeting: Minute taker: Bill. Chair: Atul Maria _______________________ Maria C. Babiuc Hamilton, Ph.D. Professor, Department of Physics College of Science, Marshall University, 1 John Marshall Drive, Huntington, WV, 25755 Room S 257, Phone: (304)696-2754

1 0

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 19 May '21

19 May '21

1 0

Compiling on DiRAC DIaL
by Elley, Matthew 19 May '21

19 May '21

Hi all, I am trying to compile the toolkit on DiRAC DIaL for which there is not currently a configuration script. I am not having issues with any external libraries except for PAPI, which is not present on the cluster. I've tried compiling with the ET PAPI bundle, but I get the following error: Cactus/arrangements/ExternalLibraries/PAPI/src/stats.c(20): catastrophic error: cannot open source file "papi.h" #include <papi.h> I cannot find the header file anywhere (i.e. in ET or /usr/include) so I am not sure how to proceed. Is PAPI a necessary library for ET function? Compiling with it excluded from the thornlist gives me /usr/bin/ld: cannot find -lsz make[1]: *** [/lustre/dirac/home/dc-elle1/ET/Cactus/exe/cactus_ET_dirac] Error 1 which I am not sure is connected as I am compiling from scratch. Thanks, Matt

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users May 2021