Users May 2023

users@lists.einsteintoolkit.org

13 participants
16 discussions

PROBLEM RUNNING GW150914
by Jose Luis Rativa Medina 06 Jul '23

06 Jul '23

Hi, I'm trying to run the GW150914 <https://einsteintoolkit.org/gallery/bbh/index.html> simulation on my MSI GF63 9sc but the parameter file contains python code and I don't know how to run it. Thanks -- *José Luis Rativa Medina* *Estudiante ing. sistemas y computación* *Universidad Nacional de Colombia* -- *Aviso legal:* El contenido de este mensaje y los archivos adjuntos son confidenciales y de uso exclusivo de la Universidad Nacional de Colombia. Se encuentran dirigidos sólo para el uso del destinatario al cual van enviados. La reproducción, lectura y/o copia se encuentran prohibidas a cualquier persona diferente a este y puede ser ilegal. Si usted lo ha recibido por error, infórmenos y elimínelo de su correo. Los Datos Personales serán tratados conforme a la Ley 1581 de 2012 y a nuestra Política de Datos Personales que podrá consultar en la página web www.unal.edu.co <http://www.unal.edu.co/>.* *Las opiniones, informaciones, conclusiones y cualquier otro tipo de dato contenido en este correo electrónico, no relacionados con la actividad de la Universidad Nacional de Colombia, se entenderá como personales y de ninguna manera son avaladas por la Universidad.

2 1

Re: [Users] Compilation Error in cluster ParamGanga
by Steven R. Brandt 19 Jun '23

19 Jun '23

I would talk to the system administrators about that. There is definitely something wrong with your compiler install if you can't find math.h. --Steve On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > Dear Steve, > > I just rechecked the cpp program to output M_PI in the computing > nodes, and that is not working right now, which was somehow working > fine a few days ago. I also tried different gcc versions (4.8, 10.2, > 11.1, gnu8/8.3) available in the cluster. All report similar issue: > > prog.cpp:1:18: fatal error: math.h: No such file or directory > #include <math.h> > ^ > compilation terminated. > > ----OR----- > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > fatal error: features.h: No such file or directory > 39 | #include <features.h> > | ^~~~~~~~~~~~ > compilation terminated. > > The cpp code seems to work fine in login node, as it gives out the > value of pi upon execution. I am contacting the hpc admin to resolve > this issue. > > I am attaching the test code (prog.cpp), the sbatch jobscript > (prog.sh) and the output file (slurn-117473.out), if you would like to > have a look at it. > > Please let me know if I should try something different to get this > working. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ᐧ > > On Mon, May 8, 2023 at 7:25 PM Shamim Haque 1910511 > <shamims(a)iiserb.ac.in> wrote: > > Thanks Steve. I'll make the changes and try again. I'll let you > know the outcome. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ᐧ > > On Mon, May 8, 2023 at 7:21 PM Steven R. Brandt > <sbrandt(a)cct.lsu.edu> wrote: > > That tends to make me think you aren't using the compiler you > think you're using. You could put full paths in the .cfg files > and recompile from scratch. > > i.e. > > CXX = /full/path/to/icc > > --Steve > > On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: >> Yes, this gives the value of Pi successfully, before starting >> the compilation: >> >> //home/vkerni.iitr/sam/trial >> 3.141593 >> Current machine: paramganga3 >> Using configuration: sim >> Reconfiguring sim >> Writing configuration to: >> /home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList >> Cactus - version: 4.12.0 >> Reconfiguring sim. >> / >> /./ >> /./ >> /./ >> /checking for M_PI... no >> configure: error: M_PI not defined. Try adding >> -D_XOPEN_SOURCE to CPPFLAGS. >> >> Error reconfiguring sim-config >> make: *** [sim-config] Error 2 >> / >> >> >> Shamim Haque >> Senior Research Fellow (SRF) >> Department of Physics >> IISER Bhopal >> >> ᐧ >> >> On Thu, May 4, 2023 at 12:29 AM Steven Brandt >> <sbrandt(a)cct.lsu.edu> wrote: >> >> So, sanity check, can you compile the following program >> outside of Cactus? >> >> #include <math.h> >> #include <stdio.h> >> >> int main() { >> printf("%f\n", M_PI); >> return 0; >> } >> >> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: >>> Dear Steve, >>> >>> I tried gcc (10 and 11) as well and I am stuck with the >>> following error: >>> /configure: error: M_PI not defined. Try adding >>> -D_XOPEN_SOURCE to CPPFLAGS./ >>> / >>> / >>> This also happened for Intel (2020 and 2019). I >>> have attached both the output files here. >>> >>> Please ignore the out file in the first email, I have >>> uploaded another out file by mistake. The rest of the >>> files are correct in the first mail. >>> >>> Regards >>> Shamim Haque >>> Senior Research Fellow (SRF) >>> Department of Physics >>> IISER Bhopal >>> >>> ᐧ >>> >>> On Wed, May 3, 2023 at 8:04 PM Steven Brandt >>> <sbrandt(a)cct.lsu.edu> wrote: >>> >>> I notice from the output that your compiler claims >>> "limits.h" is missing. Possibly there's something >>> wrong with your icc setup? Maybe you should try >>> using gcc / gfortran instead. >>> >>> --Steve >>> >>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: >>>> Dear Frank, >>>> >>>> I added the alias pattern for the computing nodes >>>> and now ETK can detect the cluster (checked by >>>> asking "whoami") during the batch queuing. I'll get >>>> in touch with the admins to include 'patch'. >>>> >>>> However, the problem still persists, having assured >>>> that the correct machine file and option list are >>>> being used during the compilation. >>>> >>>> Regards >>>> Shamim Haque >>>> Senior Research Fellow (SRF) >>>> Department of Physics >>>> IISER Bhopal >>>> >>>> ᐧ >>>> >>>> On Wed, May 3, 2023 at 3:49 PM Frank Loeffler >>>> <frank.loeffler(a)uni-jena.de> wrote: >>>> >>>> Hi Shamim >>>> >>>> On Wed, May 03, 2023 at 03:11:49PM +0530, >>>> Shamim Haque 1910511 wrote: >>>> >We are trying to compile ETK on ParamGanga at >>>> IIT Roorkee. The >>>> >compilation >>>> >stops at a very initial step and gives out the >>>> error: >>>> > >>>> >*checking whether the Fortran compiler >>>> (gfortran ) works... yeschecking >>>> >whether the Fortran compiler (gfortran ) is a >>>> cross-compiler... nochecking >>>> >whether the Fortran compiler (gfortran ) >>>> supports TYPE(*) for >>>> >CCTK_PointerTo... yeschecking how to run the C >>>> preprocessor... cppchecking >>>> >for ANSI C header files... nochecking for C99 >>>> features... yeschecking for >>>> >M_PI... noconfigure: error: M_PI not defined. >>>> Try adding -D_XOPEN_SOURCE to >>>> >CPPFLAGS.* >>>> > >>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, >>>> but it does not help. This >>>> >error is consistent if we set up using Intel >>>> Compiler (2020 or 2019), or >>>> >gcc (11 or 10). I have attached the outfile, >>>> error file, machine script, >>>> >and option lists for both cases (Intel and >>>> gcc) for any reference. >>>> > >>>> >Secondly, we are not allowed to compile ETK on >>>> login nodes. So we are >>>> >queuing a job script, which upon execution, >>>> enters the Cactus directory, >>>> >does "setup-silent", and then executes the >>>> build command. Is this the right >>>> >way to compile ETK using the batch script? Or >>>> should we do something >>>> >different? I have attached the shell script >>>> (compile.sh) used for sbatch >>>> >queue for reference. >>>> >>>> I might be wrong, but setup-silent will not use >>>> your machine file which >>>> contains all the compiler flags, but will setup >>>> something "from scratch" >>>> instead. According to the logs, this will be, >>>> e.g., in >>>> >>>> /home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini >>>> >>>> My guess would be that this misses the compiler >>>> flags necessary to have >>>> M_PI defined. The best way to get this working >>>> might be to let >>>> simfactory also detect the cluster >>>> configuration on the compute nodes >>>> (where you compile), such that setup-silent is >>>> not needed, but instead >>>> the correct cluster configuration is found and >>>> used automatically. In >>>> order to do that, look at examples of >>>> 'aliaspattern' in the mdb/machines >>>> directory. An alternative would be to tell >>>> simfactory specifically that >>>> you want to use your machine configuration file. >>>> >>>> As a side-note: I noticed the command 'patch' >>>> is missing too. This is a >>>> tool so common that it should be installed >>>> everywhere. Your admin is >>>> probably the best person to ask for advise >>>> here. It should not be too >>>> hard to install yourself, but that should not >>>> be necessary either, >>>> especially when you are told to compile on >>>> compute nodes. >>>> >>>> Frank >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)einsteintoolkit.org >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>> _______________________________________________ >>> Users mailing list >>> Users(a)einsteintoolkit.org >>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>>

3 11

Thorn setup taking too much time in cluster
by Shamim Haque 1910511 06 Jun '23

06 Jun '23

Hello, I am trying to run BNSM using IllinoisGRMHD on HPC Kanad at IISER Bhopal. While I have tested the parfile to be running fine on debug queue (1 node) and high memory queue (3 nodes), I am unable to run the simulation in a queue with 9 nodes (144 cores). The output file suggests that the setup of listed thorns is not complete within 24 hours, which is the max walltime for this queue. Is there a way to sort out this issue? I have attached the parfile and outfile for reference. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ᐧ

3 9

New Einstein Toolkit Release (Schwarzschild)
by Cupp, Samuel D. 25 May '23

25 May '23

Release Announcement Click here to read the announcement in HTML (with hyperlinks): https://einsteintoolkit.org/about/releases/ET_2023_05_announcement.html We are pleased to announce the twenty-sixth release (code name "Karl Schwarzschild") of the Einstein Toolkit, an open-source, community-developed software infrastructure for relativistic astrophysics. The major changes in this release include: * kuibit 1.4.0, which, in addition to bug fixes, improves support for working with horizon data, timer data, and higher rank objects (See release notes for more details). Two new arrangements and thorns have been added: * The FUKA reader for numerical relativity initial data * TwoPunctures_BBHSF for binary black hole with scalar fields initial data New capabilities for existing codes: * Baikal and BaikalVacuum have been updated to have READ/WRITE declarations for automated ghost zone synchronization. Several variable groups (such as RHS variables) are also no longer included in checkpoint/restart operations. * particle_tracerET can now output the particles' four-velocity (contravariant and covariant); added support for binary output format; removed limit on the number of tracer particles In addition, bug fixes accumulated since the previous release in November 2022 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community, including code to compute initial data parameters, the spacetime evolution codes Baikal, lean_public, and McLachlan, analysis codes to compute horizon characteristics and gravitational waves, the Carpet AMR infrastructure, and the relativistic magneto-hydrodynamics codes GRHydro and IllinoisGRMHD. Data analysis and post-processing is handled by the kuibit library. The Einstein Toolkit also contains a 1D self-force code. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure, providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The Einstein Toolkit uses a distributed software model, and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve staff and faculty from five different institutions and host weekly meetings that are open for anyone to join. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought-out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org, or contact the users mailing list users(a)einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 2004157/2004044/2004311/2004879/2003893 (Enabling fundamental research in the era of multi-messenger astrophysics). The Einstein Toolkit contains about 350 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. Deprecated functionality There are no features marked as deprecated in this release, to be removed in the next release. Contributors Among the many contributors to the Einstein Toolkit and to this release in particular, important contributions to new components were made by the following authors: * Cheng-Hsin Cheng * Gabriele Bozzola * Leonardo Werneck * Ludwig Jens Papenfort * Samuel Cupp * Samuel Tootle How to upgrade from Kowalevski Release (ET_2022_11) To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download.html) on the Einstein Toolkit website for download instructions. The SelfForce-1D code uses a single git repository; thus, using git pull; git checkout ET_2023_05 will update the code. To install Kuibit, do the following: pip install --user -U kuibit==1.4.0 Machine notes Supported (tested) machines include: * Debian, Ubuntu, Fedora, Mint, OpenSUSE, and macOS X installations with dependencies installed as prescribed in the official installation instructions * Deep Bayou * Delta * Expanse * Frontera * Golub * Queen Bee 2 * Queen Bee 3 * Stampede 2 * Summit * Sunrise * SuperMIC * SuperMUC * ThornyFlat Note for individual machines: * TACC machines: defs.local.ini needs to have `sourcebasedir = $WORK` and `basedir = $SCRATCH/simulations` configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * SuperMUC-NG: defs.local.ini needs to have `sourcebasedir = $HOME` and `basedir = $SCRATCH/simulations` configured for this machine. You need to determine $HOME and $SCRATCH by logging in to the machine. All repositories participating in this release carry a branch ET_2023_05 marking this release. These release branches will be updated if severe errors are found. The "Karl Schwarzschild" Release Team on behalf of the Einstein Toolkit Consortium (2023-05-24) * Leonardo Werneck * Samuel Cupp * Beyhan Karakas * Bing-Jyun Tsao * Cheng-Hsin Cheng * Jake Doherty * Konrad Topolski * Peter Diener * Roland Haas * Steven R. Brandt * Terrence Pierre Jacques * Thiago Assumpcao * Zachariah Etienne May 24, 2023

3 2

Meeting minutes for 2023-05-25
by Cupp, Samuel D. 25 May '23

25 May '23

Present: Sam, Keith, Yosef, Peter, Roland, Zach Chair: Peter Notetaker: Sam * ET_2023_05 Release is out! Peter: Gallery example now has multiple resolutions. Should we include all the data? Will increase the amount of data being stored. Roland: We could just add the data specifically for the new plot and parameter files. Peter: I've already added the parfiles and can add the new data. * SPECv8 Roland: They've asked us for the exact version number of the code they are using. As far as we know, the ball is still in their court for moving forward with this. * Tickets Sam: I still need to close the tickets associated with the release, which I'll do today. Roland: Since we have just released a new version, now is the time to start considering more serious changes to code. One change is removing the compile flags from the linker stage. This is because for compiling with CUDA the linker stage shouldn't be assuming that it is the c++ compiler. This is sufficient to compile CarpetX, but it remains to be seen whether this change will be sufficient for external libraries. Zach: What is the plan for including CarpetX? Is it still coming soon? Roland: The idea is to have it added in the November release, but this will require quite a bit of work to get CarpetX ready for a release, both with code cleanup and testing compilation on different systems. Sam: I know some of the thorns need to be renamed to allow for smooth integration with the rest of the toolkit. Hopefully this will happen sooner rather than later. Peter: Had a conversation with Zach about Ticket #963 but hasn't had a chance to look at McLachlan again since that call. Zach: I found a different version of the 'third' change which has even better results than the original. I'll share that soon. Roland: We can merge Ticket #2731 (checking for incomplete perl install). A fix is also proposed for Ticket #2526 (incorrect read/writes generated by Kranc). Ian doesn't have time to review this, so Roland will try to track down the person who added the READ/WRITE declarations to Kranc. Next week the chair is Zach and the minute taker is Roland.

1 0

HELP with error installing ET_2023_05
by crh＠panix.com 25 May '23

25 May '23

Dear Users of The Einstein Toolkit: I am installing the recent version of The Einstein Toolkit, ET_2023_05. When I run the following command git clone -b ET_2023_05 https://bitbucket.org/peterdiener/selfforce-1d.git SelfForce-1D I get the following error message: Cloning into 'SelfForce-1D'... fatal: Remote branch ET_2023_05 not found in upstream origin Would anyone know a solution to this error? Thank you. Best Regards, Charles R. Hurley Home e-mail: crh(a)panix.com <mailto:crh@panix.com>

1 0

2023 North American Einstein Toolkit School and Workshop (registration deadline approaching)
by Yosef Zlochower 22 May '23

22 May '23

Please note that the deadline to request partial travel support for the 2023 North American Einstein Toolkit School and Workshop is Thursday, June 1. The deadline for registration is June 17 The 2023 North American Einstein Toolkit School and Workshop will be held at the Rochester Institute of Technology on July 17 through July 21. This will be a hybrid conference with talks and participants both in-person and over zoom. The Einstein Toolkit is a community-driven software platform of core computational tools to advance and support research in relativistic astrophysics and gravitational physics. The conference consist of a mixture of tutorials for new users and talks about the latest developments both in the toolkit and the larger scientific community. A preliminary schedule and links to the registration page can be found at https://compact-binaries.org/content/events/2023-07-17/north-american-einst… Registration is now open and remain open until June 17, 2023. As part of the registration process, potential attendees can apply for partial travel support. In order to be eligible, applications must be received prior to June 1. Notification of partial travel support will be made by June 8.

1 0

Meeting minutes for 2023-05-18
by Steven Brandt 18 May '23

18 May '23

Present: Steve, Sam, Yosef, Keith, Roland, Zach, Peter, Jose Chair: Sam Notes: Steve Release: Most of the branches and tags have been done. Gallery: Not all done (Steve forgot). Peter did 4 different resolutions. Roland is moving the tutorial server to github Only Steve, Erik, and Roland can push to the dockerhub Steve needs to run tests on LSU machines. Issue with ticket 2497. IGMHD also has a parameter that replictaes one in TmunuBase. IGMHD will be deprecated to be replaced by GrhaylET. GrhaylET has 90% code coverage. US ET summer school: Registration is open Tutorials have been arranged EU ET summer school. Spec should still be happening. Peter has a convergence plot for NSs. Possible mug image. Steve needs to reply to email Next week: Chair: Peter, Minute Taker: Sam

1 0

GetComponents "enqueue" error
by Panagiotis Iosif 17 May '23

17 May '23

Dear ET users, I am trying to install ET on our local university cluster. I downloaded GetComponents and made it executable: curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2022_11/GetComponents chmod a+x GetComponents On the next step, when running the command: ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2022_11/einsteintoolk… I get the following error and the process stops. "Can't call method "enqueue" on an undefined value at ./GetComponents line 1040." Do you have any insight on this? Best regards, Panagiotis ------------------------------------------------------------- Dr. Panagiotis Iosif postdoctoral researcher email: piosif(a)auth.gr; paniosif(a)gmail.com Section of Astrophysics, Astronomy and Mechanics Department of Physics Aristotle University of Thessaloniki 54124, Thessaloniki, Greece -------------------------------------------------------------

2 1

meeting minutes for 2023-05-11
by Roland Haas 11 May '23

11 May '23

Present: Roland, Peter, Zach, Jose, Keith, Sam, Steve, Leo, Johnny, Yosef ET release ========== * release managers will send out draft of release statement later today * 2 weeks to release, target date May 24th * no issues with remaining inclusions * Steve is asking for images to put on the release mug, Zach suggested looking for kuibit based images * Leo is not able to assign tickets on https://trac.einsteintoolkit.org, Roland will check if there is something off with his account compared to Sam, who can ET testsuite status =================== * Peter is compiling on db1 to run TOV star * Steve will run BBH * Roland has run BNS * Johnny has run Poisson equation, will update website * Multipatch scalar wave assigned to Jake (UTAusin), runs finished, working on uploading to website * Leo reduced runtime of NRPyEllipticET test and will push change Unanswered questions ==================== * Steve will contact Shamin Haque about issues on clusters * Zach suggested creating a "unanswered tickets" lists similar to unanswered questions. Roland says this should be possible with some Python code and REST API calls to bitbucket's API. Estimated work load is about 2hrs if one knows the API. * Leo will look into Tmunu ticket in Baikal https://bitbucket.org/einsteintoolkit/tickets/issues/2497/illinoisgrmhd-is-… * WENO coefficient ticket still up for review, will be done after release https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-co… ET summer school ================ * Registration is open, email is out * Travel support can be requested at registration time CarpetX ======= * Sam is asking about existing interpolation / deaverage code in CarpetX to from cell centered averages to samples or averages at cells, faces, vertices. Roland suggests that the prolongation operators could be used for some of this, but that most code so far seems to roll their own AEILocalInterp ============== * Roland and Zach looked into C auto-generating Maple code * Roland could re-run the scripts and re-generate the code * results looks identical up to renaming of variables and pass the current ET testsuites ** Roland noticed that 3d generation log files are missing, will check what is going on * Peter has 5th order Hermite codes for field to particle interpolation in GR-SPH code, will discuss with Zach if that can be made available * coeffs derived in MMA and code exported to Fortran from there Next chair: Sam Next minute taker: Steve Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users May 2023