Users March 2019

users@lists.einsteintoolkit.org

16 participants
22 discussions

Einstein Toolkit Meeting Reminder
by Steven R. Brandt 31 Oct '19

31 Oct '19

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --Steve

8 94

Agenda for Monday's Meeting
by Steven R. Brandt 30 Oct '19

30 Oct '19

Please update the Wiki with agenda items for Monday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --Steve

3 46

New Einstein Toolkit Release (Chien-Shiung Wu)
by Steven R. Brandt 30 Oct '19

30 Oct '19

=== Release Announcement === We are pleased to announce the sixteenth release (code name "Tesla") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are Five new thorns have been added: * GiRaFFE * GiRaFFEfood * ShiftedKerrSchild * GiRaFFE_to_HydroBase * ID_converter_GiRaFFE In addition, bug fixes accumulated since the previous release in Feb 2018 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics code GRHydro. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 200 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: == Larger changes since last release == * Support has been added for GCC8. * Built-in OpenMPI tarball has been upgraded to 1.10.7 Note, we still recommend using the system MPI that is installed natively on your system. * And many bug fixes. Notably, a significant correctness problem for NewRad was addressed which caused many simulations with more than two ranks to fail https://trac.einsteintoolkit.org/ticket/2182 == New thorns or tools == * GiRaFFE (~1,600 lines of code): Solves the GRFFE equations in the formalism of Paschalidis and Shapiro [2] using the same reconstruction (PPM) and staggered vector potential methods as IllinoisGRMHD. * GiRaFFEfood (~1,300 lines of code): Code validation initial data and basic diagnostics. Contains Five 1D tests in flat spacetime (fast wave, Alfvén wave, degenerate Alfvén wave, "three-wave", FFE breakdown), One 3D test in flat spacetime (aligned rotator), and Three 3D tests in curved spacetimes ("Exact Wald" solution, "Magnetospheric Wald" solution, and split monopole). * ShiftedKerrSchild (~230 lines of code): Sets up all 3+1 quantities for a Kerr-Schild spacetime with an arbitrary radial shift, to mitigate effects of extreme spacetime curvature near r_{KS}=0. See Appendix of code release paper for full discussion. * GiRaFFE_to_HydroBase and ID_converter_GiRaFFE (~230 lines of code): Acts as a translation layer to convert 3-velocities in HydroBase's Valencia formalism to/from the 3-velocity found in the induction equation (vi = ui/u0; native to GiRaFFE). == Upcoming changes for the next releases == == New thorns planned for the next relases == * Seed_Magnetic_Fields-modified Extended Seed_Magnetic_Fields thorn for binary neutron stars. Supercedes Seed_Magnetic_Fields thorn. * Meudon_Bin_NS-modified Modifications to Meudon BNS initial data thorn to disable the overwriting of initial lapse/shift, which acts to significantly reduce coordinate eccentricity. Supercedes Meudon_Bin_NS thorn. * VolumeIntegrals_GRMHD Nice GRMHD volume integration thorn, currently depends on IllinoisGRMHD and Carpet. Performs volume integrals on arbitrary "Swiss-cheese"-like topologies, and even interoperates with Carpet to track NS centers of mass. * VolumeIntegrals_vacuum Nice GRMHD volume integration thorn, currently depends on ML_BSSN. There is a bit of code duplication and duplicated functionality between VI_GRMHD and VI_vacuum, to ensure that VI_vacuum can be used without enabling a GRMHD code. There is probably a better way of doing this, but I haven't had the time to think deeply about this. * particle_tracerET Solves the ODE \partial_t xi = vi for typically thousands of tracer particles, using an RK4 integration atop the current timestepping. E.g., one RK4 substep in the particle integration might occur every 16 RK4 substeps in the GRMHD evolution. These tracer particle positions are quite useful for visualizing magnetic field lines in a consistent way from frame-to-frame in a movie (recall that in the GRMHD approximation, the magnetic field lines stay attached the fluid elements they thread Flux Freezing?). Note that the velocity must be consistent with the velocity appearing in the GRMHD induction equation. This thorn reads in the HydroBase vel[] vector gridfunction, which assumes the Valencia formalism, and converts it into the induction equation velocity. * smallbPoynET Computes bi, b2, and three spatial components of Poynting flux. It also computes (-1-u0), which is useful for tracking unbound matter. * LeanBSSNMoL and NPScalars: LeanBSSNMoL evolves Einstein's Equations with the BSSN formulation. NPScalars computes the Newman-Penrose scalars. For the conformal factor it uses the "W" version, i.e. W=\gamma^{-1/6}, and it employs the usual "1+log" and "Gamma-driver" gauge conditions. Also available, in the "new_gauge" branch, is the shift condition of Figueras et al for their evolutions of higher dimensional BHs, see: https://arxiv.org/abs/1702.01755 https://arxiv.org/abs/1512.04532 * Proca arrangement: This repository (https://bitbucket.org/canuda/proca/src) provides the tools to evolve the Einstein-Proca system as first described in https://arxiv.org/abs/1505.00797 == How to upgrade from Tesla (ET_2018_02) == To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions. == Machine notes == Supported (tested) machines include: * Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (Homebrew and MacPorts) installations * Bluewaters * Comet * Cori * Draco * Edison * Golub * Hydra * Marconi * Minerva * Queenbee 2 * Stampede 2 * SuperMIC * Wheeler * TACC machines: defs.local.ini needs to have sourcebasedir = $WORK and basedir = $SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * A new configuration for KNL nodes is being worked on, but not yet included in the release (but compilation works and tests mostly pass). All repositories participating in this release carry a branch ET_2018_02 marking this release. These release branches will be updated if severe errors are found. The "Chien-Shiung Wu" Release Team on behalf of the Einstein Toolkit Consortium (2018-09-17) * Steven R. Brandt * Samuel D. Cupp * Peter Diener * Roland Haas * Roberto De Pietri Sep, 2018

1 1

Re: [Users] [ET Trac] [Einstein Toolkit] #1326: running loopcontrol on strange number of threads fails
by Erik Schnetter 11 Apr '19

11 Apr '19

Do you have a backtrace? -erik On Sunday, March 22, 2015, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #1326: running loopcontrol on strange number of threads fails > ------------------------------------+--------------------------------------- > Reporter: rhaas | Owner: > Type: defect | Status: new > Priority: minor | Milestone: > Component: EinsteinToolkit thorn | Version: > Resolution: | Keywords: LoopControl > ------------------------------------+--------------------------------------- > > Comment (by rhaas): > > This still happens even with current (Sun Mar 22 18:46:21 CET 2015) trunk, > though failure looks a bit different now: > {{{ > actus_sim: > /data/rhaas/postdoc/gr/ET_trunk/configs/sim/build/LoopControl/loopcontrol.cc:312: > T {anonymous}::divexact(T, T) [with T = int]: Assertion `i % j == 0' > failed > }}} > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/1326#comment:1> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Smoothed Punture thorn - Magnetised Bondi Accretion test
by Jay Vijay Kalinani 01 Apr '19

01 Apr '19

Dear all, My name is Jay V. Kalinani and I am a first year PhD student in Physics at the University of Padova, Italy. I'm currently working with Dr. Riccardo Ciolfi and Prof. Bruno Giacomazzo. I'm trying to run the Magnetised Bondi Accretion test with ET+GRHydro using the par file 'bondiM.par' (attached for your reference). This par file, located in the GRHydro_InitData repository, calls for the activation of the 'SmoothedPuncture' thorn, which I am unable to find in the Einstein Toolkit repositories. I would be grateful if you could kindly let me know where I could locate this thorn in order to use it. Thank you very much. Kind regards, Jay V. Kalinani

2 4

commit webhook for codes in the ET
by Haas, Roland 31 Mar '19

31 Mar '19

Hello Helvi, Miguel, Zach, the Einstein Toolkit code repositories currently send emails to the commits(a)einsteintoolkit.org mailing list whenever a commit is pushed to the repo. Ideally this would also happen for your repository(s) that host(s) code included in the Einstein Toolkit. To enable this, would you consider adding a web-hook. You can do so on by clicking on the "Add webhook" button in the "Webhooks" section of your repository's settings area (see attached screenshot). Once there give it some name and enter this URL in the URL field: https://einsteintoolkit.org/hooks/repo_changed.php?secret=NNN where NNN is to be taken (without the double quotes) from https://bitbucket.org/einsteintoolkit/www/src/master/hooks/repo_changed.php… (and I do not want to put the full valid URL into this emails since it gets archived into an archive which is harvestable by spambots). For now only the "Push" trigger actually does send any emails, though the others may (will) be added in the future. @Zach: this (should be, though I may have had typos) the URL you already used for one of your repos. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Multiple emails from bitbucket issues
by Ian Hinder 30 Mar '19

30 Mar '19

Hi, I've noticed that I'm getting two copies of some BitBucket issue notifications. The first is from trac(a)einsteintoolkit.org<mailto:trac@einsteintoolkit.org> because I am subscribed to that mailing list, and all the issue notifications go to it. The second is directly from BitBucket, and it is sent because I have interacted with an issue, so BitBucket assumes I am interested and "watches" the issue for me. I think the solution is to unsubscribe from trac(a)einsteintoolkit.org<mailto:trac@einsteintoolkit.org> and instead enable issue notifications from the tickets repository in bitbucket. Given this situation, is there still value in having trac@einsteintoolkit,org? I would imagine that people who want to see *all* ticket activity from the ET project will likely have a bitbucket account so they can interact with the code, and hence can enable direct bitbucket notifications. -- Ian Hinder Research Software Engineer University of Manchester, UK

2 1

New Einstein Toolkit Release (Proca)
by Steven R. Brandt 29 Mar '19

29 Mar '19

== Release Announcement == We are pleased to announce the eighteenth release (code name "Proca") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are: New arrangements and thorns have been added: * Proca - NPScalars_Proca - ProcaBase - ProcaEvolve - Proca_simpleID - TwoPunctures_KerrProca * lean_public - LeanBSSNMoL - NPScalars * wvuthorns_diagnostics - particle_tracerET - Seed_Magnetic_Fields_BNS - smallbPoynET - VolumeIntegrals_GRMHD - VolumeIntegrals_vacuum In addition, bug fixes accumulated since the previous release in September 2018 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics codes GRHydro and IllinoisGRMHD. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 400 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: == Larger changes since last release == * The Proca arrangement has been added: This repository provides the tools to * evolve the Einstein-Proca system as first described in * https://arxiv.org/abs/1505.00797. - NPScalars_Proca: Implementation of the spin-1 (electromagnetic) and spin-2 (gravitational) Newman-Penrose scalars - Proca_simpleID: Create analytic initial data for a non-rotating black hole surrounded by a Proca field with mass mu. - TwoPunctures_KerrProca: A modified TwoPunctures thorn to construct initial data for a single rotating black hole coupled to a massive vector field. * The Lean arrangement has been added: - LeanBSSNMoL: Implementation to evolve Einstein's Equations using the W-version of the BSSN formulation together with the puncture gauge. Also available, in the "new_gauge" branch, is a modified "Gamma-driver" that stabilizes highly rotating black hole spacetimes (adapted from Figueras et al; see: https://arxiv.org/abs/1512.04532). - NPScalars: Implementation of the spin-2 Newman-Penrose scalars * The WVU Diagnostics arrangement has been added: These thorns are designed * primarily to add useful diagnostics for binary neutron star simulations * performed with IllinoisGRMHD. - NSNS_parameter_files Contains parameter files for magnetized and unmagnetized BNS evolutions. - Seed_Magnetic_Fields_BNS Extended Seed_Magnetic_Fields thorn for binary neutron stars. - VolumeIntegrals_GRMHD: GRMHD volume integration thorn, currently depends on IllinoisGRMHD and Carpet. Performs volume integrals on arbitrary "Swiss-cheese"-like topologies, and even interoperates with Carpet to track NS centers of mass. - VolumeIntegrals_vacuum: Same functionality as VolumeIntegrals_GRMHD, but designed for integration of spacetime quantities. Depends on ML_BSSN and ADMBase for integrands. - particle_tracerET Solves the ODE \partial_t x^i = v^i for typically thousands of tracer particles, using an RK4 integration atop the current time stepping. - smallbPoynET Computes b^i, b^2, and three spatial components of Poynting flux. It also computes (-1-u_0), which is useful for tracking unbound matter. * Ticket tracking system moved to bitbucket: * https://bitbucket.org/einsteintoolkit/tickets/ Subversion infrastructure for * thorns is no longer maintained at LSU. Instead, the svn checkout mechanism * supported by github.com is used. Llama supports tensorweights other than 1.0 * or 0.0 Added EinsteinAnalysis/Hydro_Analysis/Hydro_Analysis_Masses.F90 in * order to compute the total baryonic mass and baryonic mass within user * defined radii. A summary of changes Carpet: - add support for very large grids where 64bit integer are needed for grid indices and sizes of transfer buffers - fix how physical_time_per_hour is computed - add functionality to align interior of grid functions to cache boundaries. This requires changes to Cactus and PUGH as well. - add a parameter "granularity" to make sure the interior of components is a multiple of N points in each direction * The version of MPI bundled with the ET is now OpenMPI 1.10.7 The * SystemTopology thorn now supports hwloc 2.0 == How to upgrade from Chien-Shiung Wu (ET_2018_09) == To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions. == Machine notes == Supported (tested) machines include: - Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (MacPorts) installations - Bluewaters - Comet - Golub - Stampede 2 - Shelob - Wheeler * TACC machines: defs.local.ini needs to have sourcebasedir = $WORK and basedir = $SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. All repositories participating in this release carry a branch ET_2019_03 marking this release. These release branches will be updated if severe errors are found. The "Proca" Release Team on behalf of the Einstein Toolkit Consortium (2019‐03‐29) * Steven R. Brandt * Samuel D. Cupp * Peter Diener * Zachariah Etienne * Roland Haas * Helvi Witek Mar, 2019

1 0

Open numbered CarpetHDF5 with visit
by Jens Mahlmann 29 Mar '19

29 Mar '19

Dear all, I have made some use of hdf5_merger and hdf5_slicer to convert my 3D simulation output into hdf5 files which only contain the data of one iteration - for example in the format var.iteration.h5 (i.e. var.*.h5). Visit recognizes this as a database, however, I am not able to use the time slider to jump between the numbered files. Is there any further conversion which I have to undertake? I would be glad to hear some best practices. Thank you very much and my best! Jens

2 1

meeting minutes for 2019-03-28
by Haas, Roland 28 Mar '19

28 Mar '19

Present: Steve, Roland, Yosef, Peter, Bill, Chris, Sam, Helvi, Zach ET release: * no update on BNS gallery example * WVU diagnostics will go in * Proca will go in * Lean will go in * bethe failing test: check if we test PETSC code on other machines, Roland has seen errors popping up on bethe for a number of releases, somewhat more frequent when oversubscribing the machine, first time that a PETSC error shows up though * draft of release notes: https://bitbucket.org/einsteintoolkit/www/src/master/about/releases/ET_2019… * will release tomorrow Friday Mar 29th, waiting for feedback on the notes ET meeting at RIT: * invites to speakers should go out today, and have announcement on the list later this week ET meeting at KCL: * room is booked for Sept 2nd to Sept 6th * will update planners and add information to website EU LISA meeting (May 13th - 15th) at AEI: * Helvi is organizer so will be there * LSU will send one person * there is some funds to support student accommodation, needs letter of supervisor Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users March 2019